ACOUSTIC SIGNAL PROCESSING METHOD, INFORMATION GENERATION METHOD, ACOUSTIC SIGNAL PROCESSING DEVICE, AND COMPUTER PROGRAM

Abstract

This acoustic signal processing method comprises: a first acquisition step for acquiring object information which includes first location information expressing the location of an object in a virtual space, first sound data expressing a first sound caused by the object, and first identification information expressing a processing means for the first sound data, and second location information which expresses the location of the listener; a first calculation step for calculating the distance between the object and the listener on the basis of the first location information included in the acquired object information, and the acquired second location information; a determination step for determining the processing means for processing the first sound data from among a first processing means for performing volume processing corresponding to the calculated distance and a second processing means for performing volume processing corresponding to a distance calculated by a method which differs from that of the first processing means, on the basis of the first identification information included in the acquired object information; a first processing step for processing the first sound data via the determined processing means; and a first output step for outputting the processed first sound data.

Description

Acoustic signal processing method, information generation method, computer program, and acoustic signal processing device

[0001] The present disclosure relates to an acoustic signal processing method and the like.

[0002] Patent Document 1 discloses a technology related to a stereophonic calculation method, which is an acoustic signal processing method, in which the volume (sound pressure) is controlled to change in inverse proportion to the distance between the sound source and the listener (observer).

[0003] JP 2013-201577 A International Publication No. 2021 / 180938

[0004] Akiyoshi Iida, Physics of Aerodynamic Noise 4: Pseudo-sound Waves and the Far Field [online], [Retrieved June 21, 2023], Internet (URL: https: / / fluid.mech.kogakuin.ac.jp / ~iida / Lectures / master / aeroacoustic.pdf) Society of Heating, Air-Conditioning and Sanitary Engineers of Japan, Practical Knowledge for Planning and Designing Air-Conditioning Equipment (Revised 4th Edition), Ohmsha, March 24, 2017, p. 236. Yoshinori Dobashi and two others, Real-time Rendering of Aerodynamic Sound Using Sound Textures Based on Computational fluid dynamics, ACM Transactions on Graphics, Vol. 22, No. 3, p732-740

[0005] However, with the technology disclosed in Patent Document 1, it may be difficult to give a sense of realism to the listener.

[0006] Therefore, an object of the present disclosure is to provide an acoustic signal processing method and the like that can give a sense of realism to listeners.

[0007] An acoustic signal processing method according to one aspect of the present disclosure includes: a first acquisition step of acquiring object information including first position information indicating a position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a processing means for the first sound data, and second position information indicating a position of a listener of the first sound in the virtual space; a first calculation step of calculating a distance between the object and the listener based on the first position information included in the acquired object information and the acquired second position information; a determination step of determining a processing means to process the first sound data from among a first processing means that processes a volume according to the calculated distance based on the first identification information included in the acquired object information and a second processing means that processes a volume according to the distance calculated in a manner different from that of the first processing means; a first processing step of processing the first sound data with the determined processing means; and a first output step of outputting the processed first sound data.

[0008] Furthermore, an information generation method according to one aspect of the present disclosure includes a second acquisition step of acquiring first sound data indicating a first sound occurring at a position related to the position of a listener in a virtual space and first position information indicating a position of an object in the virtual space, and a first generation step of generating, from the acquired first sound data and the acquired first position information, first object audio information including information about the object that causes the first sound to be reproduced at a position related to the position of the listener due to the object and the first position information.

[0009] Furthermore, an acoustic signal processing method according to one aspect of the present disclosure includes a third acquisition step of acquiring the first object audio information generated by the above-described information generation method, the acquired first sound data, and second position information indicating the position of the listener relative to the first sound; a second calculation step of calculating a distance between the object and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data such that the volume of the first sound is attenuated as the calculated distance increases; and a second output step of outputting the processed first sound data.

[0010] Furthermore, an acoustic signal processing method according to one aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the above-described information generation method, the acquired first sound data, and second position information indicating the position of the listener relative to the first sound; a second calculation step of calculating a distance between the object emitting the wind and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data such that the volume of the first sound is attenuated as the calculated distance increases; and a second output step of outputting the processed first sound data.

[0011] Moreover, an acoustic signal processing method according to one aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the above-described information generation method, the acquired first sound data, and second position information indicating the position of the listener of the first sound; a second calculation step of calculating a distance between the object emitting the wind and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of, when the calculated distance is greater than the unit distance indicated by the unit distance information included in the acquired first object audio information, processing the first sound data so as to attenuate the volume of the first sound in accordance with the calculated distance and the unit distance; and a second output step of outputting the processed first sound data.

[0012] Moreover, an acoustic signal processing method according to one aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the above-described information generation method, the acquired first sound data, and second position information indicating the position of the listener of the first sound; a second calculation step of calculating a distance between the object emitting the wind and the listener and a two-point direction connecting the object and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data so as to control a volume of the first sound based on the angle between the front direction and the calculated two-point direction and the characteristics indicated by the directivity information, and so as to attenuate a volume of the first sound in accordance with the calculated distance and the unit distance when the calculated distance is longer than the unit distance indicated by the unit distance information included in the acquired first object audio information; and a second output step of outputting the processed first sound data.

[0013] Furthermore, an acoustic signal processing method according to one aspect of the present disclosure includes: a third acquisition step of acquiring second object audio information in which the first object audio information generated by the above-described information generation method, the acquired first sound data, and second sound data indicating a second sound caused by the object are associated with the first position information; a second processing step of not convolving a first sound signal based on the acquired first sound data with a head-related transfer function that depends on the direction from which the sound arrives, and of convolving a second sound signal based on the second sound data indicated by the acquired second object audio information with a head-related transfer function that depends on the direction from which the sound arrives; and a second output step of outputting the unprocessed first sound signal and the processed second sound signal.

[0014] Furthermore, an acoustic signal processing method according to one aspect of the present disclosure includes a third acquisition step of acquiring the first object audio information generated by the above-described information generation method, the acquired first sound data, and second object audio information in which the first position information is associated with second sound data indicating a second sound caused by the object; a second processing step of processing a first sound signal based on the acquired first sound data in a manner dependent on the direction from which wind is coming and convolving a second sound signal based on the second sound data indicated by the acquired second object audio information with a head-related transfer function that is dependent on the direction from which sound is coming; and a second output step of outputting the processed first sound signal and the processed second sound signal.

[0015] Furthermore, an acoustic signal processing method according to one aspect of the present disclosure includes a third acquisition step of acquiring the first object audio information generated by the above-described information generation method, the acquired first sound data, and third object audio information in which third position information indicating the position of another object in the virtual space, the another object being different from the object, and third sound data indicating a third sound generated at the position of the other object are associated; a second processing step of processing a first sound signal based on the acquired first sound data in a manner dependent on the direction from which wind is coming, and convolving a third sound signal based on the third sound data indicated by the acquired third object audio information with a head-related transfer function that depends on the direction from which sound is coming; and a second output step of outputting the processed first sound signal and the processed third sound signal.

[0016] Moreover, an information generation method according to one aspect of the present disclosure includes a fourth acquisition step of acquiring a generation position of a first wind blowing in a virtual space, a first wind direction of the first wind, and a first assumed wind speed that is the wind speed of the first wind; a second generation step of generating fourth object audio information that associates the acquired generation position, first wind direction, and first assumed wind speed; a storage step of storing aerodynamic sound core information that includes a representative wind speed and aerodynamic sound data that indicates aerodynamic sound generated when wind at the representative wind speed reaches the ears of a listener in the virtual space; and a third output step of outputting the generated fourth object audio information and the stored aerodynamic sound core information.

[0017] Moreover, an acoustic signal processing method according to one aspect of the present disclosure includes a fifth acquisition step of acquiring the fourth object audio information and the aerodynamic sound core information output by the above-described information generation method, and second position information indicating the position of the listener in the virtual space; a third calculation step of calculating the distance between the generation position and the listener based on the generation position included in the acquired fourth object audio information and the acquired second position information; a third processing step of processing the aerodynamic sound data so that the volume of the aerodynamic sound attenuates as the calculated distance increases; and a fourth output step of outputting the processed aerodynamic sound data.

[0018] Moreover, an acoustic signal processing method according to one aspect of the present disclosure includes a fifth acquisition step of acquiring the fourth object audio information output by the above-described information generation method, the aerodynamic sound core information including data indicating the distribution of frequency components of the aerodynamic sound, and second position information indicating the position of the listener in the virtual space; a third calculation step of calculating the distance between the generation position and the listener based on the generation position included in the acquired fourth object audio information and the acquired second position information; a third processing step of processing the aerodynamic sound data so as to shift the distribution of the frequency components of the aerodynamic sound toward lower frequencies as the calculated distance increases; and a fourth output step of outputting the processed aerodynamic sound data.

[0019] Moreover, an information generation method according to one aspect of the present disclosure includes a sixth acquisition step of acquiring a second wind direction of a second wind blowing in a virtual space and a second assumed wind speed, which is the wind speed of the second wind; a third generation step of generating fifth object audio information in which the acquired second wind direction and the second assumed wind speed are associated; a storage step of storing aerodynamic sound core information including a representative wind speed and aerodynamic sound data indicating aerodynamic sound generated when wind at the representative wind speed reaches the ears of a listener in the virtual space; and a fifth output step of outputting the generated fifth object audio information and the stored aerodynamic sound core information.

[0020] Furthermore, an information generation method according to one aspect of the present disclosure includes a seventh acquisition step of acquiring a generation position of a first wind blowing in a virtual space, a first wind direction of the first wind, a first assumed wind speed which is the wind speed of the first wind, a second wind direction of a second wind blowing in the virtual space, and a second assumed wind speed which is the wind speed of the second wind; a fourth generation step of generating fourth object audio information in which the acquired generation position, the first wind direction, and the first assumed wind speed are associated, and generating fifth object audio information in which the acquired second wind direction and the second assumed wind speed are associated; and a sixth output step of outputting the generated fourth object audio information and the generated fifth object audio information.

[0021] Moreover, an acoustic signal processing method according to one aspect of the present disclosure includes an eighth acquisition step of acquiring second position information indicating the position of the listener in the virtual space, and the fourth object audio information or the fifth object audio information output by the above-described information generation method; a fourth processing step of, if the fourth object audio information is acquired, processing the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the acquired second position information, and, if the fifth object audio information is acquired, processing the aerodynamic sound data included in the aerodynamic sound core information independently of the position indicated by the acquired second position information; and a seventh output step of outputting the processed aerodynamic sound data.

[0022] Furthermore, a program according to one aspect of the present disclosure causes a computer to execute the above-described acoustic signal processing method.

[0023] Furthermore, a program according to one aspect of the present disclosure causes a computer to execute the above-described information generation method.

[0024] and a first output unit configured to output the processed first sound data. The first output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The third output unit is configured to output the processed first sound data.

[0025] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0026] According to the acoustic signal processing method according to one aspect of the present disclosure, it is possible to give a sense of realism to the listener.

[0027] FIG. 1 is a diagram illustrating a first example of aerodynamic sound (wind noise) generated as an object moves (changes in position). FIG. 2 is a diagram illustrating a second example of aerodynamic sound (wind noise) generated as an object moves (changes in position). FIG. 3 is a diagram illustrating a third example of aerodynamic sound (wind noise) generated as an object moves (changes in position). FIG. 4A is a diagram illustrating aerodynamic sound generated when wind radiating from an object reaches a listener's ear. FIG. 4B is a diagram illustrating a method for measuring the attenuation of the volume of a second aerodynamic sound depending on the distance from the wind source. FIG. 4C is a diagram illustrating the measurement results of the experiment described in FIG. 4B, showing the frequency characteristics of the picked-up sound. FIG. 4D is a diagram in which the frequency characteristics at each distance between the second aerodynamic sound and the motor sound are converted into volume and plotted for each distance. FIG. 5A is a diagram illustrating the listener's personal space. FIG. 5B is a diagram showing a stereophonic (immersive audio) reproduction system as an example of a system to which the acoustic processing or decoding processing of the present disclosure can be applied. FIG. 5C is a functional block diagram showing the configuration of an encoding device as an example of the encoding device of the present disclosure. FIG. 5D is a functional block diagram showing the configuration of a decoding device as an example of the decoding device of the present disclosure. FIG. 5E is a functional block diagram showing the configuration of an encoding device as another example of the encoding device of the present disclosure. FIG. 5F is a functional block diagram showing the configuration of a decoding device as another example of the decoding device of the present disclosure. FIG. 5G is a functional block diagram showing the configuration of a decoder as an example of the decoder in FIG. 5D or FIG. 5F. FIG. 5H is a functional block diagram showing the configuration of a decoder as another example of the decoder in FIG. 5D or FIG. 5F. FIG. 5I is a diagram showing an example of the physical configuration of an acoustic signal processing device. FIG. 5J is a diagram showing an example of the physical configuration of an encoding device. FIG. 6 is a block diagram showing the functional configuration of an acoustic signal processing device according to an embodiment. FIG. 7 is a flowchart of operation example 1 of the acoustic signal processing device according to the embodiment. FIG. 8 is a diagram showing a bat as an object and a listener according to operation example 1. Fig. 9 is a block diagram showing the functional configuration of an acoustic signal processing device according to Modification 1. Fig. 10 is a diagram showing four other people and a listener according to Modification 1.FIG. 11 is a flowchart of a second operation example of the acoustic signal processing device according to Modification 1. FIG. 12 is a block diagram showing the functional configuration of the acoustic signal processing device according to Modification 2. FIG. 13 is a diagram showing an object and a plurality of sounds according to Modification 2. FIG. 14 is a flowchart of a third operation example of the acoustic signal processing device according to Modification 2. FIG. 15 is a diagram showing a case where the object according to Modification 2 is an electric fan. FIG. 16 is a diagram showing a case where the object according to Modification 2 is a zombie. FIG. 17 is a block diagram showing the functional configuration of an information generating device and an acoustic signal processing device according to Modification 3. FIG. 18 is a diagram showing a fan as an object and a listener according to Modification 3. FIG. 19 is a diagram for explaining directionality information and unit distance information according to Modification 3. FIG. 20 is a diagram for explaining processing by the second processing unit according to Modification 3. FIG. 21 is a diagram for explaining other processing by the second processing unit according to Modification 3. FIG. 22 is a flowchart of a fourth operation example of the information generating device according to Modification 3. FIG. 23 is a flowchart of a fifth operation example of the acoustic signal processing device according to Modification 3. FIG. 24 is a block diagram showing the functional configuration of an information generating device and an acoustic signal processing device according to Modification 4. FIG. 25 is a diagram illustrating processing of first sound data according to Modification 4. FIG. 26 is a flowchart of a sixth operational example of the acoustic signal processing device according to Modification 4. FIG. 27 is a flowchart of a seventh operational example of the acoustic signal processing device according to Modification 4. FIG. 28 is a block diagram illustrating functional configurations of an information generating device and an acoustic signal processing device according to Modification 5. FIG. 29 is a flowchart of an eighth operational example of the acoustic signal processing device according to Modification 5. FIG. 30 is a block diagram illustrating functional configurations of an information generating device and an acoustic signal processing device according to Modification 6. FIG. 31 is a flowchart of a ninth operational example of the information generating device according to Modification 6. FIG. 32 is a flowchart of a tenth operational example of the acoustic signal processing device according to Modification 6. FIG. 33 is a flowchart of an eleventh operational example of the acoustic signal processing device according to Modification 6. FIG. 34 is a block diagram illustrating functional configurations of an information generating device and an acoustic signal processing device according to Modification 7. FIG. 35 is a diagram illustrating an example of an image displayed on a display unit according to Modification 7. FIG. 36 is a flowchart of a twelfth operational example of the information generating device according to Modification 7.Fig. 37 is a flowchart of an operation example 13 of an acoustic signal processing device according to Modification 7. Fig. 38 is a block diagram showing the functional configurations of an information generating device and an acoustic signal processing device according to Modification 8. Fig. 39 is a flowchart of an operation example 14 of an information generating device according to Modification 8. Fig. 40 is a functional block diagram and a diagram showing an example of steps for explaining a case where the rendering units in Figs. 5G and 5H perform pipeline processing.

[0028] (Findings that Form the Basis of the Present Disclosure) Conventionally, an acoustic signal processing method is known in which the volume (sound pressure) of a sound heard by a listener in a virtual space is controlled.

[0029] Patent Literature 1 discloses a technology related to a stereophonic calculation method, which is an acoustic signal processing method. In this acoustic signal processing method, the volume (sound pressure) is controlled so that it changes inversely proportional to the distance between a sound source and a listener (observer). More specifically, the volume is controlled so that it decreases inversely proportionally as the distance increases. This allows the listener to recognize the distance between the object emitting the sound, i.e., the sound source, and themselves.

[0030] Sounds that have been controlled in this way are used in applications for reproducing three-dimensional sounds in a space where a user (listener) is present, such as virtual reality (VR) or augmented reality (AR).

[0031] Incidentally, in real space, there are known examples of sounds in which the volume of a sound heard by a listener attenuates according to conditions other than the inverse proportion to the distance between the object emitting the sound and the listener himself.

[0032] The following two sounds are examples of such sounds.

[0033] The first type of sound is aerodynamic noise (so-called wind noise) that occurs when an object moves (see Non-Patent Document 1). Aerodynamic noise (wind noise) is sound that is based on pressure fluctuations such as air vortex streets that are generated when wind hits an object or when an object moves through the air.

[0034] FIG. 1 is a diagram illustrating a first example of aerodynamic noise (wind noise) that occurs as an object moves (changes in position). In FIG. 1, the object is a baseball bat B. When this object (bat B) moves (changes in position), that is, when the bat B is swung, wind noise is generated. By hearing this wind noise, a listener L can recognize that the bat B has been swung. In real space, the volume of this wind noise attenuates as the distance between the bat B and the listener L increases, and more specifically, it attenuates as the square of the distance between the bat B and the listener L.

[0035] If the technology disclosed in Patent Document 1 were applied to such wind noise in a virtual space, the listener L would hear wind noise that is controlled so that the volume attenuates in inverse proportion to the distance between the bat B and the listener L. In other words, the wind noise generated in a virtual space using the technology disclosed in Patent Document 1 would be different from the wind noise the listener L hears in real space. When the listener L hears wind noise generated in a virtual space using the technology disclosed in Patent Document 1, the wind noise is different from the wind noise the listener L hears in real space, causing the listener L to feel uncomfortable and making it difficult for the listener L to experience a sense of realism. Therefore, there is a need for an acoustic signal processing method or the like that can provide the listener L with a sense of realism.

[0036] FIG. 2 is a diagram illustrating a second example of aerodynamic noise (wind noise) that occurs as an object moves (changes in position). In FIG. 2, the object is an ambulance A. When this object (ambulance A) moves (changes in position), that is, when the ambulance A is traveling, wind noise is generated. By hearing this wind noise, a listener L can recognize that the ambulance A is traveling. In real space, similar to the wind noise caused by the bat B, the volume of this wind noise attenuates as the distance between the ambulance A and the listener L increases, and more specifically, it attenuates as the square of the distance between the ambulance A and the listener L.

[0037] 2, the ambulance A is also an object that emits a siren sound. In real space, the volume of this siren sound decreases in inverse proportion to the increase in the distance between the ambulance A and the listener L.

[0038] Let us consider a case where the technology disclosed in Patent Document 1 is applied to both the wind noise and siren noise of ambulance A in a virtual space.

[0039] In this case, the listener L hears a siren sound that has been controlled so that the volume attenuates in inverse proportion to the distance between the ambulance A and the listener L. In other words, the siren sound in the virtual space to which the technology disclosed in Patent Document 1 is applied sounds similar to the siren sound heard by the listener L in the real space, and the listener L is less likely to feel uncomfortable.

[0040] On the other hand, in this case, the listener L will also hear wind noise that has been controlled so that the volume attenuates in inverse proportion to the distance between the ambulance A and the listener L. In other words, the wind noise generated in the virtual space using the technology disclosed in Patent Document 1 will be different from the wind noise the listener L hears in the real space. If the listener L hears wind noise generated in the virtual space using the technology disclosed in Patent Document 1, the listener L will feel uncomfortable, and it will be difficult for the listener L to obtain a sense of realism. Therefore, there is a need for an acoustic signal processing method or the like that can provide the listener L with a sense of realism even in such cases.

[0041] As described above, the ambulance A is an object that generates a plurality of sounds (a siren sound and wind noise). Such an object is not limited to the ambulance A.

[0042] 3 is a diagram illustrating a third example of aerodynamic noise (wind noise) that occurs as an object moves (changes in position). In FIG. 3, an electric fan F is shown as an example of such an object.

[0043] When the multiple blades of this fan F move (rotate), wind noise is generated. Similar to the wind noise caused by the bat B and the wind noise caused by the ambulance A, the volume of the wind noise caused by this fan F attenuates as the distance between the fan F and the listener L increases, and more specifically, attenuates as the square of the distance between the fan F and the listener L. The fan F is also an object that emits motor sound, which is the sound made by the operation of the motor provided in the fan F. In real space, the volume of this motor sound attenuates in inverse proportion to the distance between the fan F and the listener L.

[0044] In this way, the electric fan F is also an object that generates multiple sounds (motor sound and wind sound).

[0045] Therefore, as in the case where the object is an ambulance A, if the technology disclosed in Patent Document 1 is applied to both the wind noise and motor noise of the electric fan F in a virtual space, the listener L will feel uncomfortable, and it will be difficult for the listener L to obtain a sense of realism. In particular, as shown in Figure 3, when the listener L moves from position (a) to position (b), that is, when the distance between the electric fan F and the listener L changes, the sense of discomfort becomes greater. Therefore, there is a need for an acoustic signal processing method or the like that can provide the listener L with a sense of realism.

[0046] As mentioned above, wind noise has been described as the first type of sound. Next, we will explain the second type of sound, aerodynamic sound, which is generated when wind emitted from an object reaches the ear of the listener L (see Non-Patent Documents 2 and 3). Note that the ear refers to at least one of the pinna and the outer ear.

[0047] Conventional techniques, including the technique disclosed in Patent Document 1, use object audio information. Object audio information is data in which, for example, a sound signal (sound data) indicating a sound is associated with position information indicating the position at which the sound is generated. Consider the example shown in FIG. 4A.

[0048] 4A is a diagram illustrating aerodynamic sound generated when wind W emitted from an object reaches the ear of a listener L. This aerodynamic sound, which is the second sound, is sound that is generated, for example, in accordance with the shape of the ear of the listener L, when wind W from an object, such as an electric fan F, reaches the listener L. Note that, for ease of identification, hereinafter, aerodynamic sound (wind noise) generated as the object moves (changes in position) may be referred to as the first aerodynamic sound, and the second sound, aerodynamic sound generated when wind W emitted from an object and reaching the ear of the listener L collides with the ear of the listener L, may be referred to as the second aerodynamic sound.

[0049] In Fig. 4A, the sounds caused by the object, the electric fan F, are three sounds including the first aerodynamic sound (wind noise) and motor sound, and the second aerodynamic sound, which were described in Fig. 3. Here, attention is focused on the motor sound and the second aerodynamic sound.

[0050] For example, when object audio information is used for a motor sound, a sound signal (sound data) representing the motor sound is associated with position information indicating the location where the motor sound is generated. In the object audio information, the location where the motor sound is generated is the location of the electric fan F.

[0051] Furthermore, when object audio information is used for the second aerodynamic sound, a sound signal (sound data) indicating the second aerodynamic sound is associated with position information indicating the position at which the second aerodynamic sound is generated. In the object audio information, the position at which the second aerodynamic sound is generated is the position of the listener L.

[0052] Here, the volume of the motor sound is controlled so that it attenuates in inverse proportion to the distance between the electric fan F, where the motor sound is generated, and the listener L, so that the listener L can hear the motor sound without feeling uncomfortable.

[0053] Furthermore, unlike the motor sound or the first aerodynamic sound, there is no established theory as to how the volume of the second aerodynamic sound attenuates depending on the distance between the electric fan F, which is the location where the wind W is generated, and the listener L. However, it is clear from everyday experience that the volume increases or decreases depending on the distance. Therefore, the inventors of the present application experimentally clarified the tendency for this increase or decrease using an electric fan and a dummy head microphone.

[0054] Fig. 4B is a diagram showing a method for measuring the attenuation of the volume of the second aerodynamic sound depending on the distance from the wind source (that is, the position where wind W is generated (electric fan F)). As shown in Fig. 4B, a dummy head microphone (hereinafter referred to as microphone) was placed at positions 1 m, 2 m, 4 m, etc. from the fan, and the sound generated when wind W hit it was picked up. Because the mesh on the front of fan F was removed, the only sounds picked up were the second aerodynamic sound and the motor sound, and the first aerodynamic sound caused by the mesh was excluded.

[0055] FIG. 4C shows the measurement results of the experiment described in FIG. 4B , illustrating the frequency characteristics of the collected sound. More specifically, FIG. 4C (a) shows the case where the microphone was placed at a position of 1 m, FIG. 4C (b) shows the case where the microphone was placed at a position of 2 m, and FIG. 4C (c) shows the case where the microphone was placed at a position of 4 m. The line (first line) with a large rise in frequency components below 1 kHz indicates the frequency characteristics of sound collected in the direction where the wind W hits the microphone. Therefore, the first line indicates the frequency characteristics of the combined motor sound and the second aerodynamic sound. The other line (second line) indicates the frequency characteristics of sound collected in the direction where the wind W does not hit the microphone. Therefore, the second line indicates the frequency characteristics of the motor sound alone. In both graphs, the first and second lines overlap at frequency components above approximately 1 kHz, so the frequency components of the second aerodynamic sound can be represented by the components below 1 kHz on the first line. It goes without saying that the frequency components of the motor sound can be represented by the entire second line.

[0056] FIG. 4D is a diagram in which the frequency characteristics at each distance between the second aerodynamic sound and the motor sound identified in this way are converted into volume and plotted for each distance. As is commonly believed, the motor sound tends to attenuate in proportion to the distance (inversely proportional to the first power of the distance). On the other hand, it can be seen that the second aerodynamic sound tends to attenuate at the 2.5th power of the distance (inversely proportional to the 2.5th power of the distance). (FIG. 4D also plots observation data at positions 50 cm and 3 m, which are not shown in FIG. 4B.) Therefore, by controlling the volume of the second aerodynamic sound so that it attenuates at the 2.5th power of the distance between the electric fan F, where the wind W is generated, and the listener L, the listener L can hear the second aerodynamic sound in a way that is less likely to cause discomfort. Alternatively, distance may be associated with the frequency characteristics of the second aerodynamic sound at that distance, and the frequency characteristics may be acquired using the distance as an index, and the volume may be controlled. Of course, it is known that wind speed attenuates depending on distance, so it goes without saying that the wind speed and the frequency characteristics of the second aerodynamic sound at that wind speed can be associated, and the frequency characteristics can be obtained using the wind speed calculated based on distance as an index, and the volume can be controlled.

[0057] The inventors of the present application had predicted that the exponent of the distance attenuation of the second aerodynamic sound would be 4 (inversely proportional to the fourth power of the distance). This was because they believed that the second aerodynamic sound originated from so-called cavity noise, which is aerodynamic noise generated when wind hits an object with an uneven surface. Since the volume of cavity noise is said to increase in proportion to the fourth power of the wind speed, and wind speed is said to be inversely proportional to the distance, they believed that the exponent of the distance attenuation of the second aerodynamic sound would be inversely proportional to the fourth power of the distance. However, through the above-mentioned experiments, they found that the exponent of the distance attenuation of the second aerodynamic sound was approximately 2.5. They believe this is because the pinna is not a simple cavity, but a cavity with a parabolic shape, and therefore captures the wind W more efficiently. However, because the shape of the pinna varies from person to person, the exponent cannot be limited to 2.5. However, because the second aerodynamic sound originates from cavity noise, they believe that the exponent will not exceed 4.

[0058] On the other hand, with regard to the second aerodynamic sound, the position at which the second aerodynamic sound is generated is the position of the ears of the listener L. Therefore, even if the listener L moves, the position at which the second aerodynamic sound is generated, that is, the distance between the position of the listener L and the position of the ears of the listener L, is always constant, and so the volume of the second aerodynamic sound cannot be controlled to attenuate in accordance with this distance. Therefore, if object audio information according to conventional technology is used, the listener L will end up hearing the second aerodynamic sound that gives the listener L an uncomfortable feeling. Therefore, there is a need for an acoustic signal processing method or the like that can give the listener L a sense of presence.

[0059] Furthermore, it is known that there is a personal space in the real space. Fig. 5A is a diagram illustrating the personal space of a listener L.

[0060] Personal space is the limit of how close a person (listener L in this case) can be to others, in other words, a psychological territory. This indicates that for listener L in the virtual space, there is a sense of distance between each other and listener L that cannot be expressed by physical distance alone. Personal space is classified into four categories: close distance (45 cm or less), personal distance (more than 45 cm and less than 120 cm), social distance (more than 120 cm and less than 360 cm), and public distance (more than 360 cm).

[0061] When creating a sense of distance between listener L and others in a virtual space that cannot be expressed by physical distance alone, we will consider the case where the technology disclosed in Patent Document 1 is uniformly applied to the voices of all others.

[0062] In this case, the listener L hears a voice that has been controlled so that the volume of the voice of the other person is attenuated in inverse proportion to the distance between the other person and the listener L, regardless of the relationship between the other person and the listener L. In other words, whether the other person is someone who is close to the listener L or not, the voice of the other person is controlled to be attenuated uniformly.

[0063] For this reason, it is difficult to create a personal space corresponding to the listener L and a sense of distance between the listener L and others, which cannot be expressed by physical distance alone, in a virtual space, which causes the listener L to feel uncomfortable and makes it difficult for the listener L to get a sense of realism. Therefore, there is a demand for an acoustic signal processing method or the like that can give the listener L a sense of realism.

[0064] Therefore, an acoustic signal processing method according to a first aspect of the present disclosure includes: a first acquisition step of acquiring object information including first position information indicating a position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a processing means for the first sound data, and second position information indicating a position of a listener of the first sound in the virtual space; a first calculation step of calculating a distance between the object and the listener based on the first position information included in the acquired object information and the acquired second position information; a determination step of determining a processing means to process the first sound data from among a first processing means that processes a volume according to the calculated distance based on the first identification information included in the acquired object information and a second processing means that processes a volume according to the distance calculated in a manner different from that of the first processing means; a first processing step of processing the first sound data with the determined processing means; and a first output step of outputting the processed first sound data.

[0065] As a result, the means for processing the volume of the first sound can be changed according to the first identification information, so that the first sound in the virtual space heard by the listener becomes similar to the first sound in the real space heard by the listener, more specifically, becomes a sound that reproduces the first sound in the real space. As a result, the listener is less likely to feel uncomfortable and can obtain a sense of realism. In other words, the acoustic signal processing method can provide the listener with a sense of realism.

[0066] Also, for example, an acoustic signal processing method according to a second aspect of the present disclosure is the acoustic signal processing method according to the first aspect, in which the first processing means is processing means that processes the first sound data so that the volume attenuates in inverse proportion as the calculated distance increases, and the second processing means is processing means that processes the first sound data so that the volume increases or decreases in a manner different from that of the first processing means as the calculated distance increases.

[0067] As a result, a first processing means is used that processes the first sound data so that the volume attenuates inversely as the distance increases, or a second processing means is used that increases or decreases the volume in a manner different from that of the first processing means as the distance increases, depending on the first identification information, so that the first sound in the virtual space heard by the listener becomes a sound that is more similar to the first sound in the real space heard by the listener.As a result, the listener is less likely to feel uncomfortable, and the listener can experience a greater sense of realism.In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0068] Furthermore, for example, an acoustic signal processing method according to a third aspect of the present disclosure is the acoustic signal processing method according to the first or second aspect, wherein the acquired object information includes second sound data indicating a second sound that is different from the first sound and that is caused by the object, and second identification information indicating a processing means for the second sound data; in the determining step, a processing means for processing the second sound data is determined from the first processing means and the second processing means based on the second identification information included in the acquired object information; in the first processing step, the second sound data is processed by the determined processing means; and in the first output step, the processed second sound data is output; and the object is an object associated with a plurality of sound data including the first sound data and the second sound data.

[0069] As a result, the means for processing the volume of the second sound can be changed according to the second identification information, so that the second sound in the virtual space heard by the listener also becomes similar to the second sound in the real space heard by the listener. More specifically, the volume balance between the first sound and the second sound fluctuates according to the calculated distance, similar to the volume balance occurring in the real space. This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0070] Also, for example, an acoustic signal processing method according to a fourth aspect of the present disclosure is the acoustic signal processing method according to the second aspect, in which the second processing means is processing means that processes the first sound data so that the volume attenuates as the xth power of the distance (x≠1).

[0071] Thereby, in the first processing step, a second processing means can be used that processes the first sound data so that the volume attenuates as the xth power of the distance.

[0072] Also, for example, an acoustic signal processing method according to a fifth aspect of the present disclosure is the acoustic signal processing method according to the fourth aspect, in which the first identification information indicates that the processing means for the first sound data is the second processing means and indicates the value of x.

[0073] This allows the first identification information to indicate that the processing means is the second processing means, and in the first processing step, the first sound data can be processed according to the value of x indicated by the first identification information.

[0074] Furthermore, for example, an acoustic signal processing method according to a sixth aspect of the present disclosure is the acoustic signal processing method according to the fourth aspect, wherein, when the first sound is an aerodynamic sound generated in conjunction with the movement of the object, the first identification information indicates that the processing means for the first sound data is the second processing means, and that x is α, α is a real number, and α satisfies the following formula: α>1.

[0075] As a result, in the first processing step, when the first sound is an aerodynamic sound (first aerodynamic sound), the first sound data can be processed in accordance with α, which is the value of x indicated by the first identification information.

[0076] Furthermore, for example, an acoustic signal processing method according to a seventh aspect of the present disclosure is the acoustic signal processing method according to the sixth aspect, wherein, when the first sound is an aerodynamic sound generated when wind radiated from the object reaches the ear of the listener, the first identification information indicates that the processing means for the first sound data is the second processing means, and indicates that x is β, β is a real number, and β satisfies the following equation: β>2.

[0077] As a result, in the first processing step, if the first sound is an aerodynamic sound (second aerodynamic sound) generated when wind emitted from an object reaches the listener's ear, the first sound data can be processed according to β, which is the value of x indicated by the first identification information.

[0078] Furthermore, for example, an acoustic signal processing method according to an eighth aspect of the present disclosure is the acoustic signal processing method according to the seventh aspect, in which the α and the β satisfy the following formula: α<β.

[0079] As a result, in the first processing step, the first sound data can be processed using α or β that satisfies α<β.

[0080] Also, for example, an acoustic signal processing method according to a ninth aspect of the present disclosure is the acoustic signal processing method according to the seventh or eighth aspect, including a first receiving step of receiving an operation from a user to specify the value of α or β.

[0081] As a result, in the first processing step, the first sound data can be processed using the value of α or β designated by the user.

[0082] Also, for example, an acoustic signal processing method according to a tenth aspect of the present disclosure is an acoustic signal processing method according to a second aspect, in which the first identification information is information indicating whether or not to execute the first processing means, and in the decision step, it is decided whether or not to execute the first processing means based on the acquired first identification information, and it is decided to execute the second processing means regardless of whether or not the first processing means is executed, and the second processing means is processing means that processes the first sound data so that the volume becomes a predetermined value when the calculated distance is within a predetermined threshold.

[0083] As a result, in the first processing step, the volume becomes a predetermined value only when the distance is within a predetermined threshold, thereby creating an unrealistic effect, and a second processing means can be used to process the first sound data so as to also impart a natural distance attenuation effect that occurs in reality.

[0084] Also, for example, an acoustic signal processing method according to an eleventh aspect of the present disclosure is the acoustic signal processing method according to the tenth aspect, in which the predetermined threshold is a value according to a personal space.

[0085] As a result, in the first processing step, the first sound data can be processed using a predetermined threshold value that is a value corresponding to the personal space, thereby creating a sense of psychological distance that cannot be expressed by the distance attenuation effect due to physical distance.

[0086] Also, for example, an acoustic signal processing method according to a twelfth aspect of the present disclosure is the acoustic signal processing method according to the tenth or eleventh aspect, which includes a first receiving step of receiving an operation from a user specifying that the predetermined threshold is a first specified value.

[0087] As a result, in the first processing step, the first sound data can be processed using the first designated value designated by the user.

[0088] Also, for example, an information generation method according to a thirteenth aspect of the present disclosure includes a second acquisition step of acquiring first sound data indicating a first sound occurring at a position related to the position of a listener in a virtual space and first position information indicating a position of an object in the virtual space, and a first generation step of generating, from the acquired first sound data and the acquired first position information, first object audio information including information about the object that causes the first sound to be reproduced at a position related to the position of the listener due to the object and the first position information.

[0089] This makes it possible to generate first object audio information in which the position of the object is associated with first sound data indicating a first sound caused by the object and generated at a position relative to the position of the listener. When this first object audio information is used in an acoustic signal processing method, the first sound data is processed so that the volume of the first sound decreases as the distance between the object and the listener increases. This makes the first sound in the virtual space heard by the listener similar to the first sound in the real space heard by the listener, more specifically, a sound that reproduces the first sound in the real space. This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a sense of realism. In other words, the information generation method can provide the listener with a sense of realism.

[0090] Also, for example, an information generation method according to a fourteenth aspect of the present disclosure is the information generation method according to the thirteenth aspect, in which the object emits wind, the listener is exposed to the emitted wind, and the first sound is aerodynamic sound generated when the wind emitted from the object reaches the ears of the listener.

[0091] This realizes an information generation method that can make the first sound an aerodynamic sound (second aerodynamic sound) that is generated when wind emitted from an object reaches the ears of a listener.

[0092] Also, for example, an information generation method according to a fifteenth aspect of the present disclosure is the information generation method according to the fourteenth aspect, wherein in the first generation step, the first object audio information further includes unit distance information, and the unit distance information is information including a unit distance that is a reference distance, and aerodynamic sound data that indicates the aerodynamic sound at a position that is the unit distance away from the position of the object.

[0093] This makes it possible to generate first object audio information including unit distance information. When this first object audio information is used in the acoustic signal processing method, the first sound (second aerodynamic sound) in the virtual space heard by the listener becomes a sound that is more similar to the first sound (second aerodynamic sound) in the real space heard by the listener, based on the unit distance and aerodynamic sound data. This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a greater sense of realism. In other words, the information generation method can provide the listener with a greater sense of realism.

[0094] Also, for example, an information generation method according to a sixteenth aspect of the present disclosure is the information generation method according to the fifteenth aspect, wherein in the first generation step, the first object audio information further includes directivity information, the directivity information being information that indicates characteristics according to the direction of the radiated wind, and the aerodynamic sound data indicated in the unit distance information being data that indicates the aerodynamic sound at a position that is the unit distance away from the position of the object indicated in the directivity information in a frontal direction in which the object radiates the wind.

[0095] This makes it possible to generate first object audio information including directivity information. When this first object audio information is used in the acoustic signal processing method, the first sound (second aerodynamic sound) in the virtual space heard by the listener becomes a sound that is more similar to the first sound (second aerodynamic sound) in the real space heard by the listener, based on the unit distance, aerodynamic sound data, and directivity information. This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a greater sense of realism. In other words, the information generation method can provide the listener with a greater sense of realism.

[0096] Furthermore, for example, an information generation method according to a seventeenth aspect of the present disclosure is an information generation method according to any one of the thirteenth to sixteenth aspects, in which, in the first generation step, the first object audio information is generated that further includes flag information indicating whether or not to perform a process of convolving a head-related transfer function that depends on a direction from which a sound arrives on a first sound signal that is based on the first sound data that indicates the first sound generated from the object when the first sound is played back.

[0097] This makes it possible to generate first object audio information including flag information. When this first object audio information is used in the acoustic signal processing method, the first sound in the virtual space heard by the listener may be more similar to the first sound in the real space heard by the listener because a head-related transfer function may be convolved with the first sound signal based on the first sound data. This makes it less likely that the listener will feel uncomfortable, and the listener can experience a greater sense of realism. In other words, the information generation method can provide the listener with a greater sense of realism.

[0098] Furthermore, for example, an acoustic signal processing method according to an eighteenth aspect of the present disclosure includes a third acquisition step of acquiring the first object audio information generated by the information generation method according to the thirteenth aspect, the acquired first sound data, and second position information indicating the position of the listener relative to the first sound; a second calculation step of calculating a distance between the object and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data such that the volume of the first sound is attenuated as the calculated distance increases; and a second output step of outputting the processed first sound data.

[0099] As a result, in the third acquisition step, first object audio information can be acquired in which first sound data indicating a first sound caused by the object and generated at a position relative to the listener's position is associated with the position of the object. Therefore, by processing the first sound data so that the volume of the first sound decreases as the distance between the object and the listener increases, the first sound in the virtual space heard by the listener becomes similar to the first sound in the real space heard by the listener, or more specifically, becomes a sound in which the first sound in the real space is reproduced. As a result, the listener is less likely to feel uncomfortable and can experience a sense of realism. In other words, the acoustic signal processing method can provide the listener with a sense of realism.

[0100] Furthermore, for example, an acoustic signal processing method according to a nineteenth aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method according to the fourteenth aspect, the acquired first sound data, and second position information indicating the position of the listener relative to the first sound; a second calculation step of calculating a distance between the object emitting the wind and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data such that the volume of the first sound is attenuated as the calculated distance increases; and a second output step of outputting the processed first sound data.

[0101] This realizes an acoustic signal processing method that can treat the first sound as an aerodynamic sound (second aerodynamic sound) that is generated when wind emitted from an object reaches the ears of a listener.

[0102] Furthermore, for example, an acoustic signal processing method according to a twentieth aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method according to the fifteenth aspect, the acquired first sound data, and second position information indicating the position of the listener relative to the first sound; a second calculation step of calculating a distance between the object emitting the wind and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of, when the calculated distance is greater than the unit distance indicated by the unit distance information included in the acquired first object audio information, processing the first sound data so as to attenuate the volume of the first sound in accordance with the calculated distance and the unit distance; and a second output step of outputting the processed first sound data.

[0103] As a result, in the third acquisition step, it is possible to acquire first object audio information including unit distance information. Therefore, the first sound (second aerodynamic sound) in the virtual space heard by the listener becomes a sound that is more similar to the first sound (second aerodynamic sound) in the real space heard by the listener, based on the unit distance and aerodynamic sound data. As a result, the listener is less likely to feel uncomfortable, and can experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0104] Furthermore, for example, an acoustic signal processing method according to a twenty-first aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method according to the sixteenth aspect, the acquired first sound data, and second position information indicating the position of the listener of the first sound; a second calculation step of calculating a distance between the object emitting the wind and the listener and a direction between two points connecting the object and the listener, based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data so as to control a volume of the first sound based on the angle between the front direction and the calculated direction between two points and the characteristics indicated by the directivity information, and so as to attenuate a volume of the first sound in accordance with the calculated distance and the unit distance when the calculated distance is longer than the unit distance indicated by the unit distance information included in the acquired first object audio information; and a second output step of outputting the processed first sound data.

[0105] As a result, in the third acquisition step, it is possible to acquire first object audio information including directivity information. Therefore, the first sound (second aerodynamic sound) in the virtual space heard by the listener becomes a sound that is more similar to the first sound (second aerodynamic sound) in the real space heard by the listener, based on the unit distance, the aerodynamic sound data, and the directivity information. This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0106] Furthermore, for example, an acoustic signal processing method according to a 22nd aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method according to any one of the 13th to 16th aspects, the acquired first sound data, and second object audio information in which the first position information is associated with second sound data indicating a second sound originating from the object; a second processing step of not convolving a first sound signal based on the acquired first sound data with a head-related transfer function that depends on the direction from which the sound arrives, and convolving a second sound signal based on the second sound data indicated by the acquired second object audio information with a head-related transfer function that depends on the direction from which the sound arrives; and a second output step of outputting the unprocessed first sound signal and the processed second sound signal.

[0107] As a result, the second sound in the virtual space heard by the listener becomes a sound similar to the second sound in the real space heard by the listener, more specifically, a sound in which the second sound in the real space is reproduced, because the head-related transfer function is convolved with the second sound signal based on the second sound data. This makes it less likely that the listener will feel uncomfortable, and the listener can experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0108] Furthermore, for example, an acoustic signal processing method according to a 23rd aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method according to any one of the 14th to 16th aspects, the acquired first sound data, and second object audio information in which the first position information is associated with second sound data indicating a second sound caused by the object; a second processing step of processing a first sound signal based on the acquired first sound data in a manner dependent on the direction from which wind is coming, and convolving a second sound signal based on the second sound data indicated by the acquired second object audio information with a head-related transfer function that is dependent on the direction from which sound is coming; and a second output step of outputting the processed first sound signal and the processed second sound signal.

[0109] As a result, the first sound (second aerodynamic sound) in the virtual space heard by the listener is similar to the first sound (second aerodynamic sound) in the real space heard by the listener, because the first sound signal based on the first sound data is processed in a manner that depends on the direction from which the wind is coming, and more specifically, the first sound (second aerodynamic sound) in the real space is reproduced. Furthermore, the second sound in the virtual space heard by the listener is similar to the second sound in the real space heard by the listener, because the head-related transfer function is convolved with the second sound signal based on the second sound data, and more specifically, the second sound in the virtual space is reproduced. As a result, the listener is less likely to feel uncomfortable, and the listener can experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0110] Furthermore, for example, an acoustic signal processing method according to a 24th aspect of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method according to any one of the 14th to 16th aspects, the acquired first sound data, and third object audio information in which third position information indicating the position of another object in the virtual space, the other object being different from the object, and third sound data indicating a third sound generated at the position of the other object are associated; a second processing step of processing a first sound signal based on the acquired first sound data in a manner dependent on the direction of incoming wind, and convolving a third sound signal based on the third sound data indicated by the acquired third object audio information with a head-related transfer function that depends on the direction of incoming sound; and a second output step of outputting the processed first sound signal and the processed third sound signal.

[0111] As a result, when a plurality of objects including the object and other objects are provided in a virtual space, the first sound (second aerodynamic sound) and the third sound in the virtual space heard by the listener will be similar to the first sound (second aerodynamic sound) and the third sound in the real space heard by the listener, respectively. This makes it less likely that the listener will feel uncomfortable, and the listener can experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0112] Also, for example, an information generation method according to a 25th aspect of the present disclosure includes a fourth acquisition step of acquiring the generation position of a first wind blowing in a virtual space, a first wind direction of the first wind, and a first assumed wind speed that is the wind speed of the first wind; a second generation step of generating fourth object audio information that associates the acquired generation position, first wind direction, and first assumed wind speed; a storage step of storing aerodynamic sound core information that includes a representative wind speed and aerodynamic sound data that indicates aerodynamic sound generated when wind at the representative wind speed reaches the ears of a listener in the virtual space; and a third output step of outputting the generated fourth object audio information and the stored aerodynamic sound core information.

[0113] This makes it possible to generate fourth object audio information that associates the generation position, the first wind direction, and the first expected wind speed. When this fourth object audio information is used in the acoustic signal processing method, the aerodynamic sound data is processed so that the volume of the aerodynamic sound (second aerodynamic sound) decreases as the distance between the object and the listener increases. This makes the aerodynamic sound (second aerodynamic sound) in the virtual space heard by the listener similar to the aerodynamic sound (second aerodynamic sound) in the real space heard by the listener, or more specifically, a sound that reproduces the aerodynamic sound (second aerodynamic sound) in the real space. This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a sense of realism. In other words, the information generation method can provide the listener with a sense of realism.

[0114] Also, for example, an information generation method according to a 26th aspect of the present disclosure is an information generation method according to a 25th aspect, in which the first expected wind speed is the wind speed of the first wind at a position a unit distance away from the generation position in the direction of the first wind direction, which is a reference distance.

[0115] This allows the wind speed of the first wind at a position a unit distance away to be used as the first assumed wind speed.

[0116] Also, for example, an information generation method according to a 27th aspect of the present disclosure is an information generation method according to a 26th aspect, which includes a second reception step of receiving an operation from a user specifying that the unit distance is a second specified value.

[0117] This allows the fourth object audio information to be generated using the unit distance, which is the second designated value designated by the user.

[0118] Also, for example, an information generation method according to a 28th aspect of the present disclosure is an information generation method according to a 26th or 27th aspect, which includes a second receiving step of receiving from a user an operation to specify directional information indicating characteristics according to the direction of the first wind, and in the second generating step, generates the fourth object audio information in which the acquired generation position, the first wind direction, and the first expected wind speed are associated with the directional information indicated by the received operation.

[0119] This makes it possible to generate fourth object audio information associated with the generation position, the first wind direction, the first assumed wind speed, and the directivity information designated by the user.

[0120] Also, for example, an acoustic signal processing method according to a 29th aspect of the present disclosure includes a fifth acquisition step of acquiring the fourth object audio information and the aerodynamic sound core information output by the information generation method according to any one of the 26th to 28th aspects, and second position information indicating the position of the listener in the virtual space; a third calculation step of calculating the distance between the generation position and the listener based on the generation position included in the acquired fourth object audio information and the acquired second position information; a third processing step of processing the aerodynamic sound data so that the volume of the aerodynamic sound attenuates as the calculated distance increases; and a fourth output step of outputting the processed aerodynamic sound data.

[0121] As a result, in the fifth acquisition step, it is possible to acquire fourth object audio information that is associated with the generation position, the first wind direction, and the first expected wind speed. Therefore, by processing the aerodynamic sound data so that the volume of the aerodynamic sound (second aerodynamic sound) decreases as the distance between the object and the listener increases, the aerodynamic sound (second aerodynamic sound) in the virtual space heard by the listener becomes a sound similar to the aerodynamic sound (second aerodynamic sound) in the real space heard by the listener, or more specifically, a sound that reproduces the aerodynamic sound (second aerodynamic sound) in the real space. As a result, the listener is less likely to feel uncomfortable, and the listener can obtain a sense of realism. In other words, the acoustic signal processing method can provide the listener with a sense of realism.

[0122] Furthermore, for example, an acoustic signal processing method according to a 30th aspect of the present disclosure is the acoustic signal processing method according to the 29th aspect, in which, in the third processing step, the aerodynamic sound data is processed based on an ear arrival wind speed, which is the wind speed of the first wind when it reaches the ear of the listener, and the ear arrival wind speed decreases as the calculated distance increases.

[0123] As a result, the aerodynamic sound data is processed based on the wind speed reaching the ear, so the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound that is more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0124] Also, for example, an acoustic signal processing method according to a 31st aspect of the present disclosure is the acoustic signal processing method according to the 30th aspect, in which the wind speed reaching the ear is a value that attenuates at the zth power of the value obtained by dividing the calculated distance by the unit distance.

[0125] This allows a more accurate calculation of the wind speed at the ear, so that the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). As a result, the listener is less likely to feel uncomfortable, and can obtain a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0126] Furthermore, for example, an acoustic signal processing method according to a thirty-second aspect of the present disclosure is the acoustic signal processing method according to the thirty-first aspect, in which z satisfies the following formula: z=1.

[0127] This allows a more accurate calculation of the wind speed at the ear, so that the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). As a result, the listener is less likely to feel uncomfortable, and can obtain a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0128] Furthermore, for example, an acoustic signal processing method according to a thirty-third aspect of the present disclosure is the acoustic signal processing method according to the thirty-first aspect, in which, in the third processing step, the aerodynamic sound data is processed so that the volume of the aerodynamic sound is attenuated by the γ power of the value obtained by dividing the representative wind speed by the ear wind speed.

[0129] As a result, the aerodynamic sound data is processed so that the volume of the aerodynamic sound (second aerodynamic sound) becomes a more accurate volume, and the aerodynamic sound (second aerodynamic sound) in the virtual space heard by the listener becomes a sound that more closely resembles the aerodynamic sound (second aerodynamic sound) in the real space heard by the listener. As a result, the listener is less likely to feel uncomfortable, and can obtain a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0130] Furthermore, for example, an acoustic signal processing method according to a thirty-fourth aspect of the present disclosure is the acoustic signal processing method according to the thirty-third aspect, in which the γ and z satisfy the following formula: γ×z<4.

[0131] This allows a more accurate calculation of the wind speed at the ear, so that the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). As a result, the listener is less likely to feel uncomfortable, and can obtain a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0132] Furthermore, for example, an acoustic signal processing method according to a 35th aspect of the present disclosure includes a fifth acquisition step of acquiring the fourth object audio information output by the information generation method according to any one of the 26th to 28th aspects, the aerodynamic sound core information including data indicating the distribution of frequency components of the aerodynamic sound, and second position information indicating the position of the listener in the virtual space; a third calculation step of calculating the distance between the generation position and the listener based on the generation position included in the acquired fourth object audio information and the acquired second position information; a third processing step of processing the aerodynamic sound data so as to shift the distribution of the frequency components of the aerodynamic sound toward lower frequencies as the calculated distance increases; and a fourth output step of outputting the processed aerodynamic sound data.

[0133] As a result, in the fifth acquisition step, it is possible to acquire fourth object audio information that is associated with the generation position, the first wind direction, and the first expected wind speed. Therefore, by processing the aerodynamic sound data so that the distribution of frequency components of the aerodynamic sound (second aerodynamic sound) is shifted toward lower frequencies as the distance between the object and the listener increases, the aerodynamic sound (second aerodynamic sound) in the virtual space heard by the listener becomes a sound similar to the aerodynamic sound (second aerodynamic sound) in the real space heard by the listener, or more specifically, a sound in which the aerodynamic sound (second aerodynamic sound) in the real space is reproduced. As a result, the listener is less likely to feel uncomfortable, and the listener can obtain a sense of realism. In other words, the acoustic signal processing method can provide the listener with a sense of realism.

[0134] Furthermore, for example, an acoustic signal processing method according to a 36th aspect of the present disclosure is the acoustic signal processing method according to the 35th aspect, in which, in the third processing step, the aerodynamic sound data is processed based on an ear arrival wind speed, which is the wind speed of the first wind when it reaches the ear of the listener, and the ear arrival wind speed decreases as the calculated distance increases.

[0135] As a result, the aerodynamic sound data is processed based on the wind speed reaching the ear, so the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound that is more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0136] Also, for example, an acoustic signal processing method according to a 37th aspect of the present disclosure is the acoustic signal processing method according to the 36th aspect, in which the wind speed reaching the ear is a value that attenuates at the zth power of the value obtained by dividing the calculated distance by the unit distance.

[0137] This allows a more accurate calculation of the wind speed at the ear, so that the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). As a result, the listener is less likely to feel uncomfortable, and can obtain a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0138] Furthermore, for example, an acoustic signal processing method according to a 38th aspect of the present disclosure is the acoustic signal processing method according to the 37th aspect, in which z satisfies the following formula: z=1.

[0139] This allows a more accurate calculation of the wind speed at the ear, so that the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). As a result, the listener is less likely to feel uncomfortable, and can obtain a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0140] Furthermore, for example, an acoustic signal processing method according to a thirty-ninth aspect of the present disclosure is the acoustic signal processing method according to any one of the thirty-sixth to thirty-eighth aspects, in which, in the third processing step, the aerodynamic sound data is processed so as to shift the distribution of the frequency components of the aerodynamic sound to a frequency that is an inverse multiple of the value obtained by dividing the representative wind speed by the ear wind speed.

[0141] This allows a more accurate calculation of the wind speed at the ear, so that the aerodynamic sound in the virtual space heard by the listener (second aerodynamic sound) becomes a sound more similar to the aerodynamic sound in the real space heard by the listener (second aerodynamic sound). As a result, the listener is less likely to feel uncomfortable, and can obtain a greater sense of realism. In other words, the acoustic signal processing method can provide the listener with a greater sense of realism.

[0142] Also, for example, an information generation method according to a fortieth aspect of the present disclosure includes a sixth acquisition step of acquiring a second wind direction of a second wind blowing in a virtual space and a second assumed wind speed, which is the wind speed of the second wind; a third generation step of generating fifth object audio information in which the acquired second wind direction and the second assumed wind speed are associated; a storage step of storing aerodynamic sound core information including a representative wind speed and aerodynamic sound data indicating aerodynamic sound generated when wind at the representative wind speed reaches the ears of a listener in the virtual space; and a fifth output step of outputting the generated fifth object audio information and the stored aerodynamic sound core information.

[0143] This makes it possible to generate fifth object audio information in which the second wind direction and the second estimated wind speed are associated. When this fifth object audio information is used in the acoustic signal processing method, it is possible to reproduce wind with an undetermined source (natural wind blowing outdoors), and because the aerodynamic sound data is processed without being based on the position indicated by the second position information, the aerodynamic sound caused by the second wind in the virtual space heard by the listener (second aerodynamic sound) becomes similar to the aerodynamic sound caused by the second wind in the real space heard by the listener (second aerodynamic sound), or more specifically, the aerodynamic sound caused by the second wind in the real space (second aerodynamic sound) becomes a reproduced sound. This makes it less likely that the listener will feel uncomfortable, and allows the listener to experience a sense of realism. In other words, the information generation method can provide the listener with a sense of realism.

[0144] Also, for example, an information generation method according to a forty-first aspect of the present disclosure includes a seventh acquisition step of acquiring a generation position of a first wind blowing in a virtual space, a first wind direction of the first wind, a first assumed wind speed which is the wind speed of the first wind, a second wind direction of a second wind blowing in the virtual space, and a second assumed wind speed which is the wind speed of the second wind; a fourth generation step of generating fourth object audio information in which the acquired generation position, the first wind direction, and the first assumed wind speed are associated, and generating fifth object audio information in which the acquired second wind direction and the second assumed wind speed are associated; and a sixth output step of outputting the generated fourth object audio information and the generated fifth object audio information.

[0145] This makes it possible to generate fourth object audio information in which the generation position, the first wind direction, and the first assumed wind speed are associated, and fifth object audio information in which the second wind direction and the second assumed wind speed are associated, so that two types of wind can be generated in the same virtual space: wind from which the source can be identified (such as an electric fan, an exhaust vent, or a hole), and wind from which the source cannot be identified (such as a naturally occurring breeze or a strong wind). Furthermore, when this fourth object audio information is used in an acoustic signal processing method, the aerodynamic sound data is processed based on the position indicated by the second position information, so that the aerodynamic sound caused by the first wind in the virtual space heard by the listener (second aerodynamic sound) becomes a sound similar to the aerodynamic sound caused by the first wind in the real space heard by the listener (second aerodynamic sound), or more specifically, a sound in which the aerodynamic sound caused by the first wind in the real space (second aerodynamic sound) is reproduced. Furthermore, when this fifth object audio information is used in an acoustic signal processing method, the aerodynamic sound data is processed not based on the position indicated by the second position information, so that the aerodynamic sound caused by the second wind in the virtual space heard by the listener (the second aerodynamic sound) becomes a sound similar to the aerodynamic sound caused by the second wind in the real space heard by the listener (the second aerodynamic sound), or more specifically, a sound in which the aerodynamic sound caused by the second wind in the real space (the second aerodynamic sound) is reproduced. As a result, the listener is less likely to feel uncomfortable, and can obtain a sense of realism. In other words, the information generation method can provide the listener with a sense of realism.

[0146] Also, for example, an information generation method according to a 42nd aspect of the present disclosure is an information generation method according to a 41st aspect, in which the sixth output step outputs the generated fourth object audio information when the generation position of the first wind is in the virtual space.

[0147] Thus, the information generating method can determine whether or not to output the fourth object audio information depending on the generation position.

[0148] Also, for example, an information generation method according to a 43rd aspect of the present disclosure is an information generation method according to a 42nd aspect, in which the sixth output step outputs the generated fifth object audio information if the generation position of the first wind is not in the virtual space.

[0149] Thus, the information generating method can determine whether or not to output the fifth object audio information depending on the generation position.

[0150] Also, for example, an information generation method according to a 44th aspect of the present disclosure is an information generation method according to a 43rd aspect, which includes a storage step of storing aerodynamic sound core information including a representative wind speed and aerodynamic sound data indicating aerodynamic sound generated when wind at the representative wind speed reaches the ears of a listener in the virtual space, and in the sixth output step, outputs the stored aerodynamic sound core information.

[0151] As a result, when the aerodynamic sound data included in the output aerodynamic sound core information is used in an acoustic signal processing method, the aerodynamic sound core information can be applied commonly to both the first wind and the second wind, so the memory area required to store the aerodynamic sound core information can be reduced, and moreover, the aerodynamic sound caused by the first wind (second aerodynamic sound) in the virtual space heard by the listener will be similar to the aerodynamic sound caused by the first wind (second aerodynamic sound) in the real space heard by the listener, and the aerodynamic sound caused by the second wind (second aerodynamic sound) in the virtual space heard by the listener will be similar to the aerodynamic sound caused by the second wind (second aerodynamic sound) in the real space heard by the listener. For this reason, the listener is less likely to feel uncomfortable, and can obtain a sense of realism. In other words, the information generation method can provide the listener with a sense of realism.

[0152] Also, for example, an information generation method relating to a 45th aspect of the present disclosure includes a display step of displaying an image in which a wind speed and a word expressing the wind speed are linked, and a third reception step of accepting a first operation to specify the wind speed indicated by the image displayed as the first expected wind speed, and accepting a second operation to specify the wind speed indicated by the image displayed as the second expected wind speed, and is an information generation method relating to a 44th aspect.

[0153] This allows the wind speed specified by the user to be used as the first estimated wind speed, and the wind speed specified by the user to be used as the second estimated wind speed.

[0154] Furthermore, for example, an acoustic signal processing method according to a 43rd aspect of the present disclosure includes an eighth acquisition step of acquiring second position information indicating the position of the listener in the virtual space, and the fourth object audio information or the fifth object audio information output by the information generation method according to the 44th aspect; a fourth processing step of, if the fourth object audio information has been acquired, processing the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the acquired second position information, and, if the fifth object audio information has been acquired, processing the aerodynamic sound data included in the aerodynamic sound core information without being based on the position indicated by the acquired second position information; and a seventh output step of outputting the processed aerodynamic sound data.

[0155] As a result, in the eighth acquisition step, the fourth object audio information or the fifth object audio information can be acquired. Therefore, by processing the aerodynamic sound data based on the position indicated by the second position information, the aerodynamic sound (second aerodynamic sound) caused by the first wind in the virtual space heard by the listener becomes similar to the aerodynamic sound (second aerodynamic sound) caused by the first wind in the real space heard by the listener, more specifically, the aerodynamic sound (second aerodynamic sound) caused by the first wind in the real space heard by the listener. Furthermore, by processing the aerodynamic sound data without being based on the position indicated by the second position information, the aerodynamic sound (second aerodynamic sound) caused by the second wind in the virtual space heard by the listener becomes similar to the aerodynamic sound (second aerodynamic sound) caused by the second wind in the real space heard by the listener, more specifically, the aerodynamic sound (second aerodynamic sound) caused by the second wind in the real space heard by the listener. As a result, the listener is less likely to feel uncomfortable, and can obtain a sense of realism. In other words, the acoustic signal processing method can give the listener a sense of realism.

[0156] Furthermore, a computer program according to a forty-seventh aspect of the present disclosure is a computer program for causing a computer to execute the above-described acoustic signal processing method.

[0157] This allows the computer to execute the above-described acoustic signal processing method in accordance with the computer program.

[0158] A computer program according to a forty-eighth aspect of the present disclosure is a computer program for causing a computer to execute the information generation method described above.

[0159] This allows the computer to execute the above-described information generating method in accordance with the computer program.

[0160] and a first output unit configured to output the processed first sound data. The first output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The second output unit is configured to output the processed first sound data. The third output unit is configured to output the processed first sound data. The fourth output unit is configured to output the processed first sound data.

[0161] As a result, the means for processing the volume of the first sound can be changed according to the first identification information, so that the first sound in the virtual space heard by the listener becomes similar to the first sound in the real space heard by the listener, more specifically, becomes a sound that reproduces the first sound in the real space. As a result, the listener is less likely to feel uncomfortable and can obtain a sense of realism. In other words, the acoustic signal processing device can provide the listener with a sense of realism.

[0162] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0163] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0164] The embodiments described below are all comprehensive or specific examples, and the numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the scope of the claims.

[0165] In the following description, elements may be assigned ordinal numbers such as first and second. These ordinal numbers are assigned to elements in order to identify them and do not necessarily correspond to a meaningful order. These ordinal numbers may be rearranged, newly added, or removed as appropriate.

[0166] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, the scales and the like do not necessarily match in each figure. In each figure, the same reference numerals are used to denote substantially the same components, and redundant explanations will be omitted or simplified.

[0167] In this specification, terms indicating relationships between elements such as verticality, and numerical ranges, are not expressions that only express a strict meaning, but also expressions that include a substantially equivalent range, for example, a difference of about a few percent.

[0168] 5B is a diagram showing a stereophonic (immersive audio) reproduction system A0000, which is an example of a system to which the acoustic processing or decoding processing of the present disclosure can be applied. The stereophonic reproduction system A0000 includes an acoustic signal processing device A0001 and an audio presentation device A0002.

[0169] The acoustic signal processing device A0001 performs acoustic processing on an audio signal emitted by a virtual sound source to generate an acoustically processed audio signal to be presented to a listener (i.e., a listener). The audio signal is not limited to voice, and may be any audible sound. The acoustic processing is, for example, signal processing performed on an audio signal to reproduce one or more sound-related effects that a sound generated from a sound source experiences between the time the sound is emitted and the time the listener hears the sound. The acoustic signal processing device A0001 performs the acoustic processing based on information describing factors that cause the above-mentioned sound-related effects. The spatial information includes, for example, information indicating the positions of the sound source, the listener, and surrounding objects, information indicating the shape of the space, parameters related to sound propagation, etc. The acoustic signal processing device A0001 is, for example, a PC (Personal Computer), a smartphone, a tablet, or a game console.

[0170] The acoustically processed signal is presented to a listener (user) from the audio presentation device A0002. The audio presentation device A0002 is connected to the audio signal processing device A0001 via wireless or wired communication. The audio signal after audio processing generated by the audio signal processing device A0001 is transmitted to the audio presentation device A0002 via wireless or wired communication. If the audio presentation device A0002 is composed of multiple devices, such as a device for the right ear and a device for the left ear, the multiple devices present sound in synchronization by the audio signal processing device A0001 communicating with each other or with each of the multiple devices. The audio presentation device A0002 is, for example, headphones, earphones, or a head-mounted display worn on the listener's head, or a surround speaker composed of multiple fixed speakers.

[0171] The stereophonic sound reproduction system A0000 may be used in combination with an image presentation device or a stereoscopic video presentation device that provides an ER (Extended Reality) experience, including visual AR / VR.

[0172] 5B shows an example system configuration in which the acoustic signal processing device A0001 and the audio presentation device A0002 are separate devices, but the stereophonic sound reproduction system to which the acoustic signal processing method or decoding method of the present disclosure can be applied is not limited to the configuration of FIG. 5B. For example, the acoustic signal processing device A0001 may be included in the audio presentation device A0002, which may perform both acoustic processing and sound presentation. Furthermore, the acoustic signal processing device A0001 and the audio presentation device A0002 may share the acoustic processing described in the present disclosure, or a server connected to the acoustic signal processing device A0001 or the audio presentation device A0002 via a network may perform part or all of the acoustic processing described in the present disclosure.

[0173] In the above description, the acoustic signal processing device A0001 is referred to as such, but if the acoustic signal processing device A0001 performs acoustic processing by decoding a bit stream generated by encoding at least a portion of the data of an audio signal or spatial information used for acoustic processing, the acoustic signal processing device A0001 may also be referred to as a decoding device.

[0174] (Example of Encoding Device) FIG. 5C is a functional block diagram showing the configuration of an encoding device A0100, which is an example of an encoding device of the present disclosure.

[0175] The input data A0101 is data to be coded, including spatial information and / or an audio signal, and is input to the encoder A0102. Details of the spatial information will be described later.

[0176] The encoder A0102 encodes the input data A0101 to generate encoded data A0103. The encoded data A0103 is, for example, a bit stream generated by the encoding process.

[0177] The memory A 0104 stores the encoded data A 0103. The memory A 0104 may be, for example, a hard disk or a solid-state drive (SSD), or may be other memory.

[0178] In the above description, a bitstream generated by an encoding process is given as an example of the encoded data A0103 stored in the memory A0104. However, data other than a bitstream may also be used. For example, the encoding device A0100 may convert a bitstream into a predetermined data format and store the converted data in the memory A0104. The converted data may be, for example, a file or multiplexed stream containing one or more bitstreams. Here, the file may have a file format such as ISOBMFF (ISO Base Media File Format). The encoded data A0103 may also be in the form of multiple packets generated by dividing the bitstream or file. When converting the bitstream generated by the encoder A0102 into data other than the bitstream, the encoding device A0100 may be provided with a conversion unit (not shown), or the conversion process may be performed by a CPU (Central Processing Unit).

[0179] (Example of Decoding Device) FIG. 5D is a functional block diagram showing the configuration of a decoding device A0110, which is an example of a decoding device of the present disclosure.

[0180] The memory A0114 stores, for example, the same data as the coded data A0103 generated by the coding device A0100. The memory A0114 reads out the stored data and inputs it as input data A0113 to the decoder A0112. The input data A0113 is, for example, a bitstream to be decoded. The memory A0114 may be, for example, a hard disk or an SSD, or may be some other memory.

[0181] Note that the decoding device A0110 may not directly use the data stored in the memory A0114 as the input data A0113, but may convert the read data and generate converted data as the input data A0113. The data before conversion may be, for example, multiplexed data storing one or more bitstreams. Here, the multiplexed data may be, for example, a file having a file format such as ISOBMFF. The data before conversion may also be in the form of multiple packets generated by dividing the bitstream or file. When converting data different from the bitstream read from the memory A0114 into a bitstream, the decoding device A0110 may be provided with a conversion unit (not shown), or the conversion process may be performed by a CPU.

[0182] The decoder A0112 decodes the input data A0113 to generate an audio signal A0111 that is presented to the listener.

[0183] (Another Example of Encoding Device) Fig. 5E is a functional block diagram showing the configuration of an encoding device A0120, which is another example of an encoding device of the present disclosure. In Fig. 5E, components having the same functions as those in Fig. 5C are assigned the same reference numerals as those in Fig. 5C, and descriptions of these components will be omitted.

[0184] The encoding device A0120 differs from the encoding device A0100 in that the encoding device A0100 stores the encoded data A0103 in a memory A0104, whereas the encoding device A0120 includes a transmitting unit A0121 that transmits the encoded data A0103 to the outside.

[0185] The transmitter A0121 transmits a transmission signal A0122 to another device or a server based on the encoded data A0103 or data in another data format generated by converting the encoded data A0103. The data used to generate the transmission signal A0122 is, for example, the bit stream, multiplexed data, file, or packet described in the encoding device A0100.

[0186] (Another Example of Decoding Device) Fig. 5F is a functional block diagram showing the configuration of a decoding device A0130, which is another example of a decoding device of the present disclosure. In Fig. 5F, components having the same functions as those in Fig. 5D are assigned the same reference numerals as those in Fig. 5D, and descriptions of these components will be omitted.

[0187] The decoding device A0130 differs from the decoding device A0110 in that the decoding device A0110 reads the input data A0113 from a memory A0114, whereas the decoding device A0130 includes a receiving unit A0131 that receives the input data A0113 from outside.

[0188] The receiving unit A0131 receives the received signal A0132, acquires the received data, and outputs the input data A0113 to be input to the decoder A0112. The received data may be the same as the input data A0113 to be input to the decoder A0112, or may be data in a different data format from the input data A0113. If the received data is data in a different data format from the input data A0113, the receiving unit A0131 may convert the received data into the input data A0113, or a conversion unit or CPU (not shown) included in the decoding device A0130 may convert the received data into the input data A0113. The received data is, for example, a bit stream, multiplexed data, a file, or a packet, as described in the encoding device A0120.

[0189] (Functional Description of Decoder) FIG. 5G is a functional block diagram showing the configuration of a decoder A0200, which is an example of the decoder A0112 in FIG. 5D or 5F.

[0190] The input data A0113 is an encoded bitstream, and includes encoded audio data, which is an encoded audio signal, and metadata used for acoustic processing.

[0191] The spatial information management unit A0201 acquires metadata included in the input data A0113 and analyzes the metadata. The metadata includes information describing elements that act on sounds arranged in a sound space. The spatial information management unit A0201 manages spatial information necessary for acoustic processing obtained by analyzing the metadata and provides the spatial information to the rendering unit A0203. Note that, although the information used for acoustic processing is referred to as spatial information in this disclosure, it may be referred to by other names. For example, the information used for acoustic processing may be referred to as sound space information or scene information. Furthermore, when the information used for acoustic processing changes over time, the spatial information input to the rendering unit A0203 may be referred to as a space state, a sound space state, a scene state, or the like.

[0192] Furthermore, spatial information may be managed for each sound space or for each scene. For example, when different rooms are represented as virtual spaces, each room may be managed as a scene of a different sound space, or even if the same space is represented, spatial information may be managed as different scenes depending on the situation being represented. In managing spatial information, an identifier for identifying each piece of spatial information may be assigned. The spatial information data may be included in a bitstream, which is one form of input data, or the bitstream may include an identifier for the spatial information, and the spatial information data may be acquired from a source other than the bitstream. When the bitstream includes only the identifier for the spatial information, the identifier for the spatial information may be used during rendering to acquire the spatial information data stored in the memory of the acoustic signal processing device A0001 or an external server as input data.

[0193] Note that the information managed by the spatial information management unit A0201 is not limited to information included in the bitstream. For example, the input data A0113 may include, as data not included in the bitstream, data indicating the characteristics or structure of a space acquired from a software application or server providing VR or AR. Furthermore, for example, the input data A0113 may include, as data not included in the bitstream, data indicating the characteristics or position of a listener or object. Furthermore, the input data A0113 may include, as information indicating the position of the listener, information acquired by a sensor provided in a terminal including a decoding device, or information indicating the position of the terminal estimated based on information acquired by the sensor. In other words, the spatial information management unit A0201 may communicate with an external system or server to acquire spatial information and the position of the listener. Furthermore, the spatial information management unit A0201 may acquire clock synchronization information from an external system and execute a process of synchronizing with the clock of the rendering unit A0203. Note that the space in the above description may be a virtually formed space, i.e., a VR space, or may be a real space (i.e., a physical space) or a virtual space corresponding to the real space, i.e., an AR space or an MR (Mixed Reality) space. The virtual space may also be called a sound field or a sound space. Furthermore, the information indicating a position in the above description may be information such as coordinate values ​​indicating a position within the space, information indicating a relative position with respect to a predetermined reference position, or information indicating the movement or acceleration of a position within the space.

[0194] The audio data decoder A0202 decodes the encoded audio data included in the input data A0113 to obtain an audio signal.

[0195] The encoded audio data acquired by the stereophonic sound reproduction system A0000 is a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). Note that MPEG-H 3D Audio is merely one example of an encoding method that can be used to generate the encoded audio data included in the bitstream, and the encoded audio data may also be included in a bitstream encoded in another encoding method. For example, the encoding method used may be a lossy codec such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), or Vorbis, or a lossless codec such as ALAC (Apple Lossless Audio Codec) or FLAC (Free Lossless Audio Codec), or any other encoding method may be used. For example, PCM (pulse code modulation) data may be a type of encoded audio data. In this case, the decoding process may be, for example, a process of converting an N-bit binary number into a number format (e.g., floating-point format) that can be processed by the rendering unit A0203, where the number of quantization bits of the PCM data is N.

[0196] The rendering unit A0203 receives an audio signal and spatial information as input, performs acoustic processing on the audio signal using the spatial information, and outputs an audio signal A0111 after acoustic processing.

[0197] Before starting rendering, the spatial information management unit A0201 reads metadata of the input signal, detects rendering items such as objects or sounds defined in the spatial information, and sends them to the rendering unit A0203. After starting rendering, the spatial information management unit A0201 grasps changes over time in the spatial information and the position of the listener, and updates and manages the spatial information. The spatial information management unit A0201 then sends the updated spatial information to the rendering unit A0203. The rendering unit A0203 generates and outputs an audio signal to which acoustic processing has been applied based on the audio signal included in the input data and the spatial information received from the spatial information management unit A0201.

[0198] The spatial information update process and the audio signal output process with added acoustic processing may be executed in the same thread, or the spatial information management unit A0201 and the rendering unit A0203 may be assigned to independent threads. When the spatial information update process and the audio signal output process with added acoustic processing are executed in different threads, the thread startup frequency may be set individually, or the processes may be executed in parallel.

[0199] By having the spatial information management unit A0201 and the rendering unit A0203 execute their processes in different, independent threads, computational resources can be preferentially allocated to the rendering unit A0203. Therefore, in the case of sound output processing in which even the slightest delay cannot be tolerated, for example, sound output processing in which a delay of even one sample (0.02 msec) would cause a popping noise, can be safely performed. In this case, the allocation of computational resources to the spatial information management unit A0201 is limited. However, compared to audio signal output processing, updating spatial information is a less frequent process (e.g., a process such as updating the listener's facial orientation). Therefore, unlike audio signal output processing, updating spatial information does not necessarily require an instantaneous response, and therefore limiting the allocation of computational resources does not significantly affect the acoustic quality experienced by the listener.

[0200] The spatial information may be updated periodically at preset times or intervals, or when preset conditions are met. The spatial information may be updated manually by a listener or a sound space manager, or may be triggered by a change in an external system. For example, if a listener operates a controller to instantly warp the position of their avatar, instantly advance or reverse the time, or if a virtual space manager suddenly changes the environment of the venue, the thread in which the spatial information management unit A0201 is located may be activated as a one-off interrupt process in addition to being activated periodically.

[0201] The role of the information update thread that executes the spatial information update process is, for example, to update the position or orientation of the listener's avatar placed in the virtual space based on the position or orientation of the VR goggles worn by the listener, and to update the position of objects moving in the virtual space. These tasks are handled within a processing thread that runs relatively infrequently, on the order of several tens of Hz. Processing to reflect the properties of direct sound may be performed in such an infrequently occurring processing thread. This is because the properties of direct sound change less frequently than the frequency of audio processing frames for audio output. Doing so can relatively reduce the computational load of the process and also avoid the risk of pulsive noise occurring when information is updated at an unnecessarily fast frequency.

[0202] FIG. 5H is a functional block diagram showing the configuration of a decoder A0210, which is another example of the decoder A0112 in FIG. 5D or 5F.

[0203] Figure 5H differs from Figure 5G in that the input data A0113 includes an unencoded audio signal rather than encoded audio data. The input data A0113 includes a bitstream including metadata and an audio signal.

[0204] The spatial information management unit A0211 is the same as the spatial information management unit A0201 in FIG. 5G, and therefore a description thereof will be omitted.

[0205] The rendering unit A0213 is the same as the rendering unit A0203 in FIG. 5G, and therefore a description thereof will be omitted.

[0206] In the above description, the configuration in Figure 5H is called a decoder, but it may also be called an audio processing unit that performs audio processing. Also, a device including an audio processing unit may be called an audio processing device rather than a decoding device. Also, the audio signal processing device A0001 may be called an audio processing device.

[0207] (Physical configuration of audio signal processing device) Fig. 5I is a diagram showing an example of the physical configuration of an audio signal processing device. Note that the audio signal processing device in Fig. 5I may be a decoding device. Furthermore, part of the configuration described here may be provided in the audio presentation device A0002. Furthermore, the audio signal processing device shown in Fig. 5I is an example of the audio signal processing device A0001 described above.

[0208] The acoustic signal processing device of FIG. 5I includes a processor, a memory, a communication IF, a sensor, and a speaker.

[0209] The processor may be, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in memory to perform the acoustic processing or decoding processing of the present disclosure. Furthermore, the processor may be a dedicated circuit that performs signal processing on an audio signal, including the acoustic processing of the present disclosure.

[0210] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0211] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The acoustic signal processing device shown in FIG. 5I has a function of communicating with other communication devices via the communication IF and acquires a bitstream to be decoded. The acquired bitstream is stored in a memory, for example.

[0212] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0213] The sensor performs sensing to estimate the position or orientation of the listener. Specifically, the sensor estimates the position and / or orientation of the listener based on one or more detection results of the position, orientation, movement, velocity, angular velocity, or acceleration of a part or the entire body of the listener, such as the head, and generates position information indicating the position and / or orientation of the listener. Note that the position information may be information indicating the position and / or orientation of the listener in real space, or information indicating a displacement of the position and / or orientation of the listener based on the position and / or orientation of the listener at a predetermined time. Furthermore, the position information may be information indicating the position and / or orientation relative to the stereophonic sound reproduction system or an external device equipped with the sensor.

[0214] The sensor may be, for example, an imaging device such as a camera or a ranging device such as LiDAR (Light Detection and Ranging), and may capture an image of the listener's head movement and detect the movement of the listener's head by processing the captured image. Alternatively, the sensor may be a device that performs position estimation using wireless signals of any frequency band, such as millimeter waves.

[0215] The audio signal processing device shown in FIG. 5I may acquire position information from an external device equipped with a sensor via a communication IF. In this case, the audio signal processing device may not include a sensor. Here, the external device may be, for example, the audio presentation device A0002 described in FIG. 5B or a 3D video playback device worn on the listener's head. In this case, the sensor may be a combination of various sensors, such as a gyro sensor and an acceleration sensor.

[0216] The sensor may, for example, detect the angular velocity of rotation around at least one of three mutually perpendicular axes in the sound space as the axis of rotation as the speed of movement of the listener's head, or may detect the acceleration of displacement with at least one of the three axes as the direction of displacement.

[0217] The sensor may detect, for example, the amount of movement of the listener's head by detecting the amount of rotation about at least one of three mutually orthogonal axes in the sound space as the rotation axis, or by detecting the amount of displacement about at least one of the three axes as the displacement direction. Specifically, the sensor detects 6 DoF (position (x, y, z) and angle (yaw, pitch, roll)) as the position of the listener. The sensor is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor.

[0218] The sensor may be any device capable of detecting the position of the listener, such as a camera or a GPS (Global Positioning System) receiver. Alternatively, the sensor may use location information obtained by performing self-position estimation using LiDAR (Laser Imaging Detection and Ranging). For example, when the audio signal reproduction system is implemented by a smartphone, the sensor is built into the smartphone.

[0219] The sensor may also include a temperature sensor such as a thermocouple that detects the temperature of the acoustic signal processing device shown in Figure 5I, and a sensor that detects the remaining charge of a battery provided in or connected to the acoustic signal processing device.

[0220] A speaker has, for example, a diaphragm, a drive mechanism such as a magnet or a voice coil, and an amplifier, and presents an audio signal after acoustic processing to a listener as sound. The speaker operates the drive mechanism in response to the audio signal (more specifically, a waveform signal indicating the waveform of the sound) amplified by the amplifier, and the drive mechanism vibrates the diaphragm. In this way, the diaphragm vibrating in response to the audio signal generates sound waves, which propagate through the air and reach the listener's ears, causing the listener to perceive the sound.

[0221] Note that, although the description has been given here of an example in which the acoustic signal processing device shown in FIG. 5I includes a speaker and presents an acoustically processed audio signal via the speaker, the means for presenting the audio signal is not limited to the above configuration. For example, the acoustically processed audio signal may be output to an external audio presentation device A0002 connected via a communication module. Communication via the communication module may be wired or wireless. As another example, the acoustic signal processing device shown in FIG. 5I may include a terminal for outputting an analog audio signal, and a cable such as earphones may be connected to the terminal to present the audio signal from the earphones. In the above case, the audio signal is reproduced by headphones, earphones, a head-mounted display, a neck speaker, a wearable speaker, a surround speaker composed of multiple fixed speakers, or the like, which are worn on the head or part of the body of the listener, which is the audio presentation device A0002.

[0222] (Physical Configuration of Encoding Device) Fig. 5J is a diagram showing an example of the physical configuration of an encoding device. The encoding device shown in Fig. 5J is an example of the encoding devices A0100 and A0120 described above.

[0223] The encoding device of FIG. 5J includes a processor, a memory, and a communication IF.

[0224] The processor may be, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and the encoding process of the present disclosure may be performed by the CPU or GPU executing a program stored in memory. Alternatively, the processor may be a dedicated circuit that performs signal processing on an audio signal, including the encoding process of the present disclosure.

[0225] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0226] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The encoding device has a function of communicating with other communication devices via the communication IF and transmits an encoded bitstream.

[0227] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0228] [Configuration] First, a description will be given of the configuration of the acoustic signal processing device 100 according to the embodiment. Fig. 6 is a block diagram showing the functional configuration of the acoustic signal processing device 100 according to the present embodiment.

[0229] The acoustic signal processing device 100 according to this embodiment is a device for processing and outputting first sound data indicating a first sound caused by an object in a virtual space (sound reproduction space). The acoustic signal processing device 100 according to this embodiment is a device that is used in various applications in virtual spaces, such as virtual reality or augmented reality (VR / AR), for example.

[0230] Objects in a virtual space are not particularly limited as long as they are included in the content displayed on the display unit 30 that displays content (here, video as an example) executed within the virtual space. The objects are moving objects, and include, for example, objects representing plants and animals, man-made objects, or natural objects. Examples of objects representing man-made objects include vehicles, bicycles, and airplanes. Examples of objects representing man-made objects include sports equipment such as baseball bats and tennis rackets, furniture such as desks, chairs, fans, and grandfather clocks, and buildings such as apartment buildings and commercial facilities. Note that, for example, an object may be at least one of an object that can move within the content and an object that can be moved, but is not limited to this. Note that, for example, a fan F shown in FIGS. 3 and 4A is placed on the floor, and the blades of the fan F move (rotate) even if the fan F itself does not move. Such a fan F is also included in the object.

[0231] The first sound is a sound caused by an object. For example, the first sound is a sound generated by the object. More specifically, the first sound is as follows.

[0232] An example of the first sound according to the present embodiment is aerodynamic sound (wind noise) that occurs in virtual space as an object moves, and is the first aerodynamic sound. The wind noise is a sound that represents a vortex street of air that is generated when wind W hits an object.

[0233] An example of the first sound according to this embodiment is aerodynamic sound (second aerodynamic sound) that is generated in the virtual space when wind W emitted from an object reaches the ears of a listener L. The second aerodynamic sound is sound that is generated, for example, according to the shape of the ears of the listener L, when wind W from an electric fan F, which is an object, reaches the listener L. More specifically, the second aerodynamic sound is sound caused by wind W that is generated by the movement of air due to the movement of the object.

[0234] The acoustic signal processing device 100 generates first sound data indicating a first sound in a virtual space and outputs the first sound data to the headphones 20 .

[0235] Next, the headphones 20 will be described.

[0236] The headphones 20 are a device that reproduces the first sound and are an audio output device. More specifically, the headphones 20 reproduce the first sound based on the first sound data output by the acoustic signal processing device 100. This allows the listener L to hear the first sound. Note that instead of the headphones 20, other output channels such as speakers may be used.

[0237] As shown in FIG. 6, the headphones 20 include a head sensor unit 21 and an output unit 22 .

[0238] The head sensor unit 21 senses the position of the listener L, which is determined by the horizontal coordinates and vertical height in the virtual space, and outputs second position information indicating the position of the listener L of the first sound in the virtual space to the acoustic signal processing device 100.

[0239] The head sensor unit 21 may sense information about 6 DoF (Degrees of Freedom) of the head of the listener L. For example, the head sensor unit 21 may be an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetic sensor, or a combination thereof.

[0240] The output unit 22 is a device that reproduces a sound that reaches the listener L in a sound reproduction space. More specifically, the output unit 22 reproduces the first sound based on first sound data that indicates the first sound that has been processed by the acoustic signal processing device 100 and output from the acoustic signal processing device 100.

[0241] Next, the display unit 30 will be described.

[0242] The display unit 30 is a display device that displays content (video) including objects in a virtual space. The process by which the display unit 30 displays content will be described later. The display unit 30 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro Luminescence) panel, for example.

[0243] Next, the acoustic signal processing device 100 shown in FIG. 6 will be described.

[0244] As shown in Figure 6, the acoustic signal processing device 100 includes a first acquisition unit 110, a first calculation unit 120, a determination unit 130, a first processing unit 140, a first output unit 150, a first reception unit 160, and a memory unit 170.

[0245] The first acquisition unit 110 acquires object information and second position information. The object information is information including first position information indicating the position of the object in the virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a processing means for the first sound data, that is, information about the object. Note that the object information may also include shape information indicating the shape of the object. As described above, the second position information is information indicating the position of the listener L in the virtual space. The first acquisition unit 110 may acquire the object information and the second position information from, for example, an input signal, or may acquire the object information and the second position information from other sources. The input signal will be described below.

[0246] The input signal may be composed of, for example, spatial information, sensor information, and sound data (audio signal). The above information and sound data may be included in a single input signal, or may be included in multiple separate signals. The input signal may include a bitstream composed of sound data and metadata (control information), in which case the metadata may include information identifying the spatial information and sound data.

[0247] The first position information, the second position information, the shape information, and the feasibility information described above may be included in the input signal. More specifically, the first information, the shape information, and the feasibility information may be included in spatial information, and the second information may be generated based on information acquired from sensor information. The sensor information may be acquired from the head sensor unit 21 or from another external device.

[0248] Spatial information is information about the sound space (three-dimensional sound field) created by the stereophonic sound reproduction system, and is composed of information about objects included in the sound space and information about the listener. Objects include sound source objects that emit sound and act as sound sources, and non-sound-emitting objects that do not emit sound. Non-sound-emitting objects function as obstacle objects that reflect sounds emitted by sound source objects, but sound source objects may also function as obstacle objects that reflect sounds emitted by other sound source objects. Obstacle objects may also be called reflecting objects.

[0249] Information commonly assigned to sound source objects and non-sound-producing objects includes position information, shape information, and the rate of attenuation of the volume when the object reflects sound.

[0250] The position information is expressed as coordinate values ​​on three axes, for example, the X-axis, Y-axis, and Z-axis, in Euclidean space, but does not necessarily have to be three-dimensional information. The position information may also be two-dimensional information expressed as coordinate values ​​on two axes, for example, the X-axis and Y-axis. The position information of an object is determined by a representative position of a shape expressed by a mesh or voxels.

[0251] The shape information may include information about the surface material.

[0252] The attenuation rate may be expressed as a real number equal to or less than 1 or equal to or greater than 0, or may be expressed as a negative decibel value. Since sound volume is not amplified by reflection in real space, a negative decibel value is set for the attenuation rate. However, for example, to create an eerie feeling in an unreal space, an attenuation rate of equal to or greater than 1, i.e., a positive decibel value, may be set. Furthermore, different values ​​for the attenuation rate may be set for each frequency band constituting multiple frequency bands, or values ​​may be set independently for each frequency band. Furthermore, if an attenuation rate is set for each type of material on the object surface, a corresponding attenuation rate value may be used based on information about the surface material.

[0253] Furthermore, the information commonly assigned to the sound source object and the non-sound-producing object may include information indicating whether the object belongs to a living thing or not, information indicating whether the object is a moving object or not, etc. If the object is a moving object, the position information may move over time, and the changed position information or the amount of change is transmitted to the rendering units A0203 and A0213.

[0254] The information about the sound source object includes, in addition to the information commonly assigned to the sound source object and the non-sound generating object described above, sound data and information necessary for radiating the sound data into the sound space. The sound data is data representing the sound perceived by the listener, including information about the frequency and intensity of the sound. The sound data is typically a PCM signal, but may also be data compressed using an encoding method such as MP3. In this case, the signal must be decoded at least before reaching the generation unit (the generation unit 907 described later in FIG. 40 ). Therefore, the rendering units A0203 and A0213 may include a decoding unit (not shown). Alternatively, the signal may be decoded by the audio data decoder A0202.

[0255] At least one piece of sound data may be set for one sound source object, but multiple pieces of sound data may also be set. Furthermore, identification information for identifying each piece of sound data may be assigned, and the identification information for the sound data may be stored as metadata as information about the sound source object.

[0256] The information necessary to radiate sound data into a sound space may include, for example, information on the reference volume that serves as a reference when playing the sound data, information on the position of the sound source object, information on the orientation of the sound source object, and information on the directionality of the sound emitted by the sound source object.

[0257] The reference volume information may be, for example, the effective value of the amplitude value of the sound data at the sound source position when the sound data is radiated into a sound space, and may be expressed as a floating-point decibel (db) value. For example, when the reference volume is 0 db, the reference volume information may indicate that sound is radiated into the sound space from the position indicated by the information regarding the position at the same volume without increasing or decreasing the volume of the signal level indicated by the sound data. When the reference volume information is -6 db, the reference volume information may indicate that sound is radiated into the sound space from the position indicated by the information regarding the position with the volume of the signal level indicated by the sound data reduced to about half. The reference volume information may be assigned to one piece of sound data or to multiple pieces of sound data collectively.

[0258] The volume information included in the information necessary to radiate sound data into a sound space may include, for example, information indicating time-series fluctuations in the volume of the sound source. For example, if the sound space is a virtual conference room and the sound source is a speaker, the volume transitions intermittently over a short period of time. Expressed more simply, this can be said to mean that sound portions and silent portions alternate. Furthermore, if the sound space is a concert hall and the sound source is a performer, the volume is maintained for a certain period of time. Furthermore, if the sound space is a battlefield and the sound source is an explosive, the volume of the explosion increases for only a moment and then remains silent. In this way, the volume information of the sound source includes not only information about the volume of the sound but also information about the transition of the volume of the sound, and such information may be used as information indicating the properties of the sound data.

[0259] Here, the information on loudness transitions may be data indicating frequency characteristics in a time series. The information on loudness transitions may be data indicating the duration of a section in which sound is present. The information on loudness transitions may be data indicating a time series of the duration of a section in which sound is present and the duration of a section in which sound is absent. The information on loudness transitions may be data listing, in a time series, multiple sets of durations during which the amplitude of a sound signal can be considered steady (considered to be roughly constant) and data on the amplitude values ​​of the signal during those periods. The information on loudness transitions may be data indicating the durations during which the frequency characteristics of a sound signal can be considered steady. The information on loudness transitions may be data listing, in a time series, multiple sets of durations during which the frequency characteristics of a sound signal can be considered steady and data on the frequency characteristics during those periods. The data format of the information on loudness transitions may be, for example, data indicating the outline of a spectrogram. Furthermore, the volume that serves as a reference for the frequency characteristics may be the reference volume. The information on the reference volume and the information indicating the properties of the sound data may be used to calculate the volume of the direct sound or reflected sound to be perceived by the listener, as well as in a selection process to select whether or not to perceive it.

[0260] Orientation information is typically expressed using yaw, pitch, and roll. Alternatively, the roll rotation may be omitted and the information may be expressed using azimuth (yaw) and elevation (pitch). Orientation information may change over time, and if it does change, it is transmitted to the rendering units A0203 and A0213.

[0261] The information about the listener is information about the position and orientation of the listener in sound space. The position information is expressed as a position on the XYZ axes in Euclidean space, but it does not necessarily have to be three-dimensional information and may be two-dimensional information. The information about orientation is typically expressed using yaw, pitch, and roll. Alternatively, the information about orientation may be expressed using azimuth (yaw) and elevation (pitch) without the roll rotation. The position information and orientation information may change over time, and if they change, they are transmitted to the rendering units A0203 and A0213.

[0262] The sensor information includes the amount of rotation or displacement detected by a sensor worn by the listener, as well as the position and orientation of the listener. The sensor information is transmitted to the rendering units A0203 and A0213, which update the position and orientation information of the listener based on the sensor information. The sensor information may be, for example, position information obtained by a mobile terminal performing self-position estimation using a GPS, a camera, or LiDAR (Laser Imaging Detection and Ranging). Information obtained from an external source other than a sensor via a communication module may also be detected as sensor information. Information indicating the temperature of the acoustic signal processing device 100 and information indicating the remaining battery level may be acquired from the sensor. Information indicating the computing resources (CPU capacity, memory resources, PC performance) of the acoustic signal processing device 100 or the audio presentation device A0002 may be acquired in real time as sensor information.

[0263] In the present embodiment, the first acquisition unit 110 acquires the object information from the storage unit 170, but is not limited to this, and may acquire the object information from a device other than the acoustic signal processing device 100 (for example, a server device 10 such as a cloud server). Furthermore, the first acquisition unit 110 acquires the second position information from the headphones 20 (more specifically, the head sensor unit 21), but is not limited to this.

[0264] Here, the information contained in the object information will be described.

[0265] First, the first location information will be described.

[0266] As described above, the object in the virtual space is included in the content (image) displayed on the display unit 30, and may be at least one of an object that can be moved within the content and an object that can be moved. For example, the object in the virtual space is the bat B shown in Figure 1.

[0267] The first position information is information indicating the position of bat B in the virtual space at a given point in time. Note that in the virtual space, bat B may move when swung. For this reason, the first acquisition unit 110 continuously acquires the first position information. The first acquisition unit 110 acquires the first position information, for example, each time the spatial information is updated by the spatial information management units A0201 and A0211.

[0268] Next, the first sound data indicating the first sound will be described.

[0269] The sound data including the first sound data described in this specification may be a sound signal such as PCM (Pulse Code Modulation) data, but is not limited to this and may be any information that indicates the properties of the sound.

[0270] As an example, if a sound signal is a noise signal with a volume of X decibels, the sound data related to the sound data may be the PCM data itself representing the sound signal, or may be data consisting of information indicating that the component is a noise signal and information indicating that the volume is X decibels. As another example, if a sound signal is a noise signal with a frequency component having a predetermined peak / dip characteristic, the sound data related to the sound data may be the PCM data itself representing the sound signal, or may be data consisting of information indicating that the component is a noise signal and information indicating the peak / dip of the frequency component.

[0271] In this specification, a sound signal based on sound data means PCM data representing the sound data.

[0272] The first identification information will be further described.

[0273] The first identification information is information indicating a processing means for the first sound data. That is, in this embodiment, a first processing means and a second processing means are provided as processing means for the first sound data. The first processing means and the second processing means both process the volume of the first sound indicated by the first sound data, and process the first sound data in mutually different manners. The first identification information is information indicating that the processing means for the first sound data is the first processing means, the second processing means, or both the first processing means and the second processing means.

[0274] The type of processing means indicated by the first identification information is predetermined according to the object indicated by the first identification information. For example, the type of processing means indicated by the first identification information is predetermined by the creator of the content (video) displayed on the display unit 30.

[0275] Next, the shape information will be described.

[0276] The shape information is information that indicates the shape of an object (e.g., bat B) in virtual space. The shape information indicates the shape of the object, and more specifically, indicates the three-dimensional shape of the object as a rigid body. The shape of the object is indicated by, for example, a sphere, a rectangular parallelepiped, a cube, a polyhedron, a cone, a pyramid, a cylinder, a prism, or a combination thereof. Note that the shape information may be expressed as, for example, mesh data, or as, for example, a voxel, a three-dimensional point cloud, or a set of multiple faces each consisting of vertices having three-dimensional coordinates.

[0277] In addition, the first position information includes object identification information for identifying the object, the first identification information also includes object identification information for identifying the object, and the shape information also includes object identification information for identifying the object.

[0278] Therefore, even if the first acquisition unit 110 acquires the first position information, the first sound data, the first identification information, and the shape information separately, the object indicated by each of the first position information, the first sound data, the first identification information, and the shape information can be identified by referencing the object identification information included in each of the first position information, the first sound data, the first identification information, and the shape information. For example, in this example, it is easy to identify that the object indicated by each of the first position information, the first sound data, the first identification information, and the shape information is the same bat B. In other words, by referencing the four pieces of object identification information, it becomes clear that the first position information, the first sound data, the first identification information, and the shape information acquired by the first acquisition unit 110 are information related to bat B. Therefore, the first position information, the first sound data, the first identification information, and the shape information are linked as information indicating bat B.

[0279] Next, the second location information will be described.

[0280] The listener L can move in the virtual space. The second position information is information indicating the position of the listener L in the virtual space at a given time. Since the listener L can move in the virtual space, the first acquisition unit 110 continuously acquires the second position information. The first acquisition unit 110 acquires the first position information, for example, each time the spatial information is updated by the spatial information management units A0201 and A0211.

[0281] The first position information, first sound data, first identification information, shape information, etc. may be included in metadata, control information, or header information included in the input signal. When the first sound data is a sound signal (PCM data), information identifying the sound signal may be included in the metadata, control information, or header information, or the sound signal may be included in information other than the metadata, control information, or header information. In other words, the acoustic signal processing device 100 (more specifically, the first acquisition unit 110) may acquire metadata, control information, or header information included in the input signal and perform acoustic processing based on the metadata, control information, or header information. The acoustic signal processing device 100 (more specifically, the first acquisition unit 110) may acquire the first position information, first sound data, first identification information, and shape information described above, and the source of acquisition is not limited to the input signal. The first sound data and metadata may be stored in a single input signal, or may be stored separately in multiple input signals.

[0282] Furthermore, the input signal may contain audio signals other than the first sound data as audio content information. The audio content information may be encoded using MPEG-H 3D Audio (ISO / IEC 23008-3) (hereinafter referred to as MPEG-H 3D Audio) or other encoding techniques. The encoding technique is not limited to MPEG-H 3D Audio, and other known techniques may also be used. Information such as the above-mentioned first position information, first sound data, first identification information, and shape information may also be subject to encoding.

[0283] In other words, the acoustic signal processing device 100 acquires the sound signal and metadata contained in the encoded bitstream. The acoustic signal processing device 100 acquires and decodes audio content information. In this embodiment, the acoustic signal processing device 100 functions as a decoder included in a decoding device, and more specifically, functions as rendering units A0203 and A0213 included in the decoder. Note that the term "audio content information" in the present disclosure is to be interpreted as information including the sound signal itself, first position information, first sound data, first identification information, and shape information, depending on the technical content.

[0284] The second location information may also be encoded. That is, the first acquisition unit 110 acquires and decodes the second location information.

[0285] The first acquisition unit 110 outputs the acquired object information and second position information to the first calculation unit 120 and the determination unit 130 .

[0286] The first calculation unit 120 calculates the distance between the object (e.g., bat B) and the listener L based on the first position information included in the object information acquired by the first acquisition unit 110 and the acquired second position information. As described above, the first acquisition unit 110 acquires the first position information and the second position information in the virtual space each time the space information is updated by the space information management units A0201 and A0211. The first calculation unit 120 calculates the distance between the object and the listener L in the virtual space based on the multiple pieces of first position information and the multiple pieces of second position information acquired each time the space information is updated. The first calculation unit 120 outputs the calculated distance between the object and the listener L to the determination unit 130.

[0287] The determination unit 130 determines which of the first processing unit and the second processing unit is to process the first sound data, based on the first identification information included in the object information acquired by the first acquisition unit 110. The first processing unit is a processing unit that processes the volume according to the distance calculated by the first calculation unit 120. The second processing unit is a processing unit that processes the volume according to the distance calculated by the first calculation unit 120 in a manner different from that of the first processing unit.

[0288] As described above, the first identification information indicates a processing means for the first sound data, and the determination unit 130 determines a processing means for processing the first sound data in accordance with the processing means indicated by the first identification information. For example, when the first identification information indicates the first processing means as the processing means for the first sound data, the determination unit 130 determines that the processing means for processing the first sound data is the first processing means.

[0289] The first processing unit 140 processes the first sound data using the processing means determined by the determination unit 130. For example, when the determination unit 130 determines that the processing means that processes the first sound data is the first processing means, the first processing unit 140 processes the first sound data using the first processing means.

[0290] The first output unit 150 outputs the first sound data processed by the first processing unit 140. Here, the first output unit 150 outputs the first sound data to the headphones 20. This allows the headphones 20 to play back the first sound indicated by the output first sound data.

[0291] The first receiving unit 160 receives an operation from a user of the acoustic signal processing device 100 (e.g., a creator of content executed in a virtual space). Specifically, the first receiving unit 160 is implemented by a hardware button, but may also be implemented by a touch panel or the like. The operation may include an operation for specifying a file name. The file may be a file in which object information including the above-described first position information, first sound data, first identification information, and shape information is formatted according to a predetermined rule. The first receiving unit 160 may receive the object information by deformatting the file. This is not limited to the first receiving unit 160, but also applies to a receiving unit 41, a second receiving unit 51, and a third receiving unit 61, which will be described later.

[0292] The memory unit 170 is a storage device that stores computer programs executed by the first acquisition unit 110, the first calculation unit 120, the determination unit 130, the first processing unit 140, the first output unit 150, and the first reception unit 160, as well as object information.

[0293] Here, the shape information according to the present embodiment will be explained again. The shape information is information used to generate an image of an object in a virtual space and is also information indicating the shape of the object (bat B). In other words, the shape information is also information used to generate content (images) to be displayed on the display unit 30.

[0294] The first acquisition unit 110 also outputs the acquired shape information to the display unit 30. The display unit 30 acquires the shape information output by the first acquisition unit 110. The display unit 30 further acquires attribute information indicating attributes (such as color) of the object (bat B) other than the shape in the virtual space. The display unit 30 may acquire the attribute information directly from a device other than the acoustic signal processing device 100 (the server device 10), or may acquire the attribute information from the acoustic signal processing device 100. The display unit 30 generates and displays content (video) based on the acquired shape information and attribute information.

[0295] The first processing means and the second processing means will be explained again.

[0296] The first processing means processes the first sound data so that the volume attenuates in inverse proportion to the increase in the distance calculated by the first calculation unit 120 .

[0297] If the distance is D and the volume of the first sound processed by the first processing means is V1, V1 is expressed by (Equation 1).

[0298] V1 ∝ (1 / D)...(Formula 1)

[0299] Furthermore, the second processing means processes the first sound data so that the volume increases or decreases in a manner different from that of the first processing means as the distance calculated by the first calculation unit 120 increases. As an example, the second processing means processes the first sound data so that the volume attenuates at the xth power of the distance (x≠1). In this case, if the volume of the first sound processed by the second processing means is V2, V2 is expressed by (Equation 2). Note that "^" in (Equation 2) represents an operator for calculating a power.

[0300] V2 ∝ (1 / D)^x... (Formula 2)

[0301] Furthermore, in this embodiment, the first identification information is also information indicating that the processing means for the first sound data is the second processing means, and indicating the value of x. More specifically, if the first sound is aerodynamic sound generated as an object moves, that is, first aerodynamic sound (wind noise), the first identification information indicates that the processing means for the first sound data is the second processing means, and that the above x is α, which is a real number and satisfies (Equation 3).

[0302] α>1...(Formula 3)

[0303] The first accepting unit 160 accepts an operation to specify the value of α from the user of the acoustic signal processing device 100. As a result, for example, x indicated in the first identification information included in the object information stored in the storage unit 170 becomes α. In other words, x indicated in the first identification information is updated to become α.

[0304] Furthermore, although the first processing unit and the second processing unit are described herein as processing the first sound data according to (Equation 1) and (Equation 2), respectively, the present invention is not limited thereto. For example, V3 calculated as V3 = f(d^r) (r is a real number equal to or greater than 1) using a function f with an argument d, which is the distance calculated by the first calculation unit 120, may be used as the volume. A monotonically decreasing function such as an inverse proportional function is often used as this function f, but a function other than a monotonically decreasing function may be used in consideration of the balance between realism in the real space and special effects in the virtual space (e.g., immersion and entertainment value). The function to be used is set in advance by the system designer. Alternatively, multiple functions may be prepared in advance, and the multiple functions may be switched between using parameters such as object information or the object's position, or the listener's state or position as indicators.

[0305] Furthermore, instead of using the function f, a structure having a table of volume V4 relative to distance d may be used. The type of table to be used can be freely designed by the system designer, taking into consideration the balance between realism in the real space and special effects in the virtual space (for example, immersiveness and entertainment value). The table to be used is set in advance by the system designer. Alternatively, multiple tables may be prepared in advance, and the multiple functions may be switched and used using parameters such as object information or object position, and the state or position of the listener as indicators.

[0306] Hereinafter, a first operational example of the acoustic signal processing method performed by the acoustic signal processing device 100 will be described.

[0307] 7 is a flowchart of an operation example 1 of the acoustic signal processing device 100 according to this embodiment. FIG. 8 is a diagram showing a bat B, which is an object, and a listener L according to the operation example 1.

[0308] In this operation example, the first sound is a sound caused by the bat B, which is an object, and is a first aerodynamic sound (wind noise) that occurs as the bat B moves.

[0309] 7, first, the first receiving unit 160 receives an operation to specify the value of α, which is an example of x, indicated in the first identification information (S10). This step S10 corresponds to a first receiving step.

[0310] As a result, x indicated in the first identification information included in the object information stored in storage unit 170 becomes α.

[0311] Furthermore, the first acquisition unit 110 acquires object information including the first location information, the first sound data, and the first identification information, and the second location information (S20). This step S20 corresponds to the first acquisition step.

[0312] Next, the first calculation unit 120 calculates the distance D between the object and the listener L based on the first position information included in the object information acquired by the first acquisition unit 110 and the second position information acquired by the first acquisition unit 110 (S30). That is, here, the first calculation unit 120 calculates the distance D between the bat B and the listener L. This step S30 corresponds to the first calculation step.

[0313] Next, the determination unit 130 determines which of the first processing unit and the second processing unit is to process the first sound data, based on the first identification information included in the object information acquired by the first acquisition unit 110 (S40). This step S40 corresponds to the determination step.

[0314] Next, the first processing unit 140 processes the first sound data using the processing means determined by the determination unit 130 (S50). This step S50 corresponds to the first processing step.

[0315] For example, if it is determined in step S40 that the processing means is the first processing means, the first processing unit 140 processes the first sound data using the first processing means. Also, for example, if it is determined in step S40 that the processing means is the second processing means, the first processing unit 140 processes the first sound data using the second processing means. Here, it is assumed that it is determined in step S40 that the processing means is the second processing means, and that the second processing means processes the first sound data so that the volume attenuates at the xth power of the distance D. Since x is α in step S10, if the volume of the first sound processed by the second processing means is V2, V2 is expressed by (Equation 4).

[0316] V2 ∝ (1 / D)^α... (Formula 4)

[0317] For example, α is 2.

[0318] The first output unit 150 outputs the first sound data processed by the first processing unit 140 (S60). This step S60 corresponds to a first output step.

[0319] In this embodiment, the means for processing the volume of the first sound can be changed according to the first identification information, so that the first sound in the virtual space heard by the listener L is similar to the first sound in the real space heard by the listener L, more specifically, is a reproduced version of the first sound in the real space. This makes it difficult for the listener L to feel uncomfortable, and allows the listener L to experience a sense of realism.

[0320] Furthermore, in this embodiment, the first sound is a first aerodynamic sound (wind noise). Also, a first processing means is used that processes the first sound data in accordance with the first identification information so that the volume attenuates in inverse proportion as the distance D increases, or a second processing means is used that increases or decreases the volume in a manner different from that of the first processing means as the distance D increases. Therefore, the first sound (wind noise) in the virtual space heard by the listener L is more similar to the first sound (wind noise) in the real space heard by the listener L. Therefore, the listener L is less likely to feel uncomfortable, and the listener L can experience a greater sense of realism.

[0321] (Modification 1 of the embodiment) Modification 1 of the embodiment will be described below. The following description will focus on the differences from the embodiment, and the description of the commonalities will be omitted or simplified.

[0322] [Configuration] First, a configuration of an acoustic signal processing device 100a according to Modification 1 of the present embodiment will be described. Fig. 9 is a block diagram showing the functional configuration of the acoustic signal processing device 100a according to this modification.

[0323] The acoustic signal processing device 100a according to this modification can create, in a virtual space, a personal space corresponding to the listener L and a sense of distance between the listener L and others that cannot be expressed by physical distance alone. The acoustic signal processing device 100a has the same configuration as the acoustic signal processing device 100 according to the embodiment, except that it includes a first processing unit 140a instead of the first processing unit 140.

[0324] In this modified example, the objects in the virtual space are other people other than the listener L. Fig. 10 is a diagram showing four other people A1 to D1 and the listener L according to this modified example. In Fig. 10, each of the four other people A1 to D1 is classified into one of four categories of personal space: close distance, personal distance, social distance, and public distance.

[0325] For example, let us say that the degree of intimacy with listener L decreases in the order of stranger A1, stranger B1, stranger C1, and stranger D1. Listener L can tolerate stranger A1 approaching up to a close distance, stranger B1 approaching up to a personal distance, stranger C1 approaching up to a social distance, and stranger D1 approaching up to a public distance.

[0326] In this modification, when the object is another person A1, the first sound is the voice of another person A1. Similarly, when the object is another person B1, the first sound is the voice of another person B1, when the object is another person C1, the first sound is the voice of another person C1, and when the object is another person D1, the first sound is the voice of another person D1.

[0327] The first acquisition unit 110 included in the acoustic signal processing device 100a acquires object information and second position information corresponding to other people corresponding to the object. There is object information corresponding to each of the four other people A1 to D1, and the object information corresponding to other person A1 may be referred to as object information A1. Similarly, the object information corresponding to other person B1 may be referred to as object information B1, the object information corresponding to other person C1 may be referred to as object information C1, and the object information corresponding to other person D1 may be referred to as object information D1. Note that when it is not necessary to distinguish between each of the object information A1 to D1, they may be simply referred to as object information.

[0328] The first processing unit 140a according to this modification processes the first sound data using the processing means determined by the determination unit 130. Here, first, second, and third example processing methods performed by the first processing unit 140a will be described.

[0329] First Example First, a processing method of the first example will be described.

[0330] In the first example, the second processing means according to this modification is processing means that processes the first sound data so that the volume becomes a predetermined value when the distance D calculated by the first calculation unit 120 is within a predetermined threshold value. This predetermined threshold value is a value according to the personal space.

[0331] That is, the second processing means processes the first sound data so that the volume heard by the listener L becomes a predetermined value according to the distance D. The predetermined value indicates, for example, a value VH indicating a higher volume and a value VL indicating a lower volume than VH. More specifically, VH is a volume that is high enough to annoy the listener L that someone else is approaching, and VL is a volume that makes the listener L aware that someone else is approaching.

[0332] First, in this modified example, the second processing means will be described in the case where the object is a stranger, such as stranger A1, who is classified into the close distance category. Listener L is close to stranger A1 and allows stranger A1 to approach him / her up to a close distance (45 cm or less). Therefore, the second processing means processes the first sound data so that the volume of the voice of stranger A1 is VL when distance D is 45 cm or less, and so that the volume attenuates as distance D increases when distance D is greater than 45 cm.

[0333] Next, the second processing means will be described when the object is a stranger classified into the personal distance category, such as stranger B1. Listener L is somewhat close to stranger B1 and allows stranger B1 to approach up to a personal distance (more than 45 cm and less than 120 cm). Therefore, the second processing means processes the first sound data so that the volume of the voice of stranger B1 is VH when distance D is 45 cm or less, VL when distance D is more than 45 cm and less than 120 cm, and attenuates as distance D increases when distance D is greater than 120 cm.

[0334] Next, the second processing means will be described when the object is a stranger classified into the social distance category, such as stranger C1. Listener L is somewhat unfamiliar with stranger C1 and allows him or her to approach up to a social distance (greater than 120 cm and less than 350 cm). Therefore, the second processing means processes the first sound data so that the volume of the voice of stranger C1 is VH when distance D is 120 cm or less, VL when distance D is 120 cm or more and less than 350 cm, and attenuates as distance D increases when distance D is greater than 350 cm.

[0335] Next, the second processing means will be described for the case where the object is a stranger classified into the public distance category, such as stranger D1. The listener L is not close to stranger D1 and allows stranger D1 to approach up to the public distance (350 cm). Therefore, the second processing means processes the first sound data so that the volume of the voice of stranger D1 is VH when the distance D is 350 cm or less, and attenuates as the distance D increases when the distance D is greater than 350 cm.

[0336] It is preferable that the category of the personal space to which each of the other people A1 to D1 is classified is indicated in the first identification information included in the corresponding object information A1 to D1.

[0337] Furthermore, as described above, the predetermined threshold is a value according to the personal space, but for example, the first receiving unit 160 may receive an operation from the user specifying that the predetermined threshold is a first specified value, and the first specified value specified by the received operation may become the predetermined threshold.

[0338] Second Example Next, a processing method of a second example will be described.

[0339] In the second example, before the processing of the first processing unit 140a is performed, the decision unit 130 decides whether to execute the first processing means based on the first identification information acquired by the first acquisition unit 110, and decides to execute the second processing means regardless of whether the first processing means is executed. Note that in the second example, the first identification information is information that indicates whether to execute the first processing means, but is information that does not indicate whether to execute the second processing means.

[0340] That is, the first processing unit 140a executes the first processing means on the first sound data when the determination unit 130 determines to execute the first processing means, and does not execute the first processing means on the first sound data when the determination unit 130 determines not to execute the first processing means. Furthermore, regardless of whether the first processing means has been executed on the first sound data, the first processing unit 140a executes the second processing means on the first sound data.

[0341] In the second example, the second processing means according to this modification is a processing means that processes the first sound data so that the volume becomes a predetermined value when the distance D calculated by the first calculation unit 120 is within a predetermined threshold value. This predetermined threshold value is a value according to the personal space.

[0342] In the following, an example will be described in which both the first processing means and the second processing means are executed on the first sound data.

[0343] First, the first processing means is executed, and the first sound data is processed so that the sound attenuates in inverse proportion to the increase in distance D, regardless of whether the distance D between the object (e.g., another person B1) and the listener L is within a predetermined threshold. Then, the second processing means is executed on the first sound data processed by the first processing means. The second processing means in the second example processes the first sound data so that the volume of the voice of the other person B1 is VH when the distance D is 45 cm, and so that the volume of the voice of the other person B1 is VL when the distance D is 120 cm.

[0344] Therefore, the first sound data executed by the first processing means and the second processing means attenuates inversely as the distance D increases, and indicates that if the distance D is 45 cm, the volume of the voice of the other person B1 is VH, and if the distance D is 120 cm, the volume of the voice of the other person B1 is VL.

[0345] In addition, the second processing means in the second example is a processing means that processes the first sound data so that when the object is another person A1 and the distance D is 45 cm, the volume of the voice of the other person A1 becomes VL.

[0346] In addition, the second processing means in the second example is a processing means that processes the first sound data so that when the object is another person C1, the volume of the voice of the other person C1 is VH if the distance D is 120 cm, and so that the volume of the voice of the other person C1 is VL if the distance D is 350 cm.

[0347] In addition, the second processing means in the second example is a processing means that processes the first sound data so that when the object is another person D1 and the distance D is 350 cm, the volume of the voice of the other person D1 becomes VH.

[0348] <Third Example> Next, a processing method of a third example will be described.

[0349] In the third example, the second processing means according to this modification is processing means that processes the first sound data so that the volume attenuates at the xth power (x≠1) of the distance D. The closer the listener L is to another person, the smaller the value of x relating to that other person becomes.

[0350] For example, if the object is a stranger classified in the close distance category such as stranger A1, the value of x is 0.9, if the object is a stranger classified in the personal distance category such as stranger B1, the value of x is 1.5, if the object is a stranger classified in the social distance category such as stranger C1, the value of x is 2.0, and if the object is a stranger classified in the public distance category such as stranger D1, the value of x is 3.0.

[0351] In all of the above first to third examples, various variables are set so that when an unfamiliar person approaches, the voice volume is increased to make the person feel annoyed, but conversely, the volume may be set to be decreased because people do not want to hear an annoying voice. In other words, while the first processing means is intended to reproduce a physical phenomenon, the second processing means may be used to produce an increase or decrease in volume depending on psychological distance that cannot be expressed by physical distance.

[0352] A second operational example of the acoustic signal processing method performed by the acoustic signal processing device 100a will now be described.

[0353] [Operation Example 2] FIG. 11 is a flowchart of operation example 2 of the acoustic signal processing device 100a according to this modification.

[0354] 11 , first, the first accepting unit 160 accepts an operation from the user specifying that the predetermined threshold is a first designated value (S11). As an example, the predetermined threshold is a value according to the personal space. Note that in the case of the third example above, this step S11 does not need to be performed.

[0355] Furthermore, the first acquisition unit 110 acquires object information including the first position information, the first sound data, and the first identification information, and the second position information (S20). The object information acquired in step S20 is at least one of the four pieces of object information A1 to D1.

[0356] Next, the first calculation unit 120 calculates the distance D between the object (other person) and the listener L based on the first position information contained in the object information acquired by the first acquisition unit 110 and the second position information acquired by the first acquisition unit 110 (S30).

[0357] Next, the determination unit 130 determines, based on the first identification information contained in the object information acquired by the first acquisition unit 110, which of the first processing means and the second processing means will process the first sound data (S40).

[0358] Next, the first processing unit 140a processes the first sound data using the processing means determined by the determination unit 130 (S50). As described above, the first sound data is processed using one of the methods in the first to third examples.

[0359] The first output unit 150 outputs the first sound data processed by the first processing unit 140a (S60).

[0360] For example, in this modification, if an unfamiliar stranger D1 approaches to within 350 cm of the listener L, the listener L will hear the voice of the stranger D1 at a volume of VH that the listener finds annoying. On the other hand, if a close stranger A1 approaches to within 45 cm of the listener L, the listener L will hear the voice of the stranger A1 at a volume of VL that the listener feels is close. In other words, the volume is controlled so that the voice of the unfamiliar stranger D1 is annoying, and the voice of the close stranger A1 is audible. Therefore, in this modification, it is possible to create in a virtual space a personal space corresponding to the listener L and a sense of distance between the listener L and others that cannot be expressed by physical distance alone.

[0361] To summarize, in this modification, the first processing step can use a second processing means that processes the first sound data so that the volume becomes a predetermined value when the distance D is within a predetermined threshold. Furthermore, the first processing step can process the first sound data using a predetermined threshold that is a value corresponding to the personal space. Therefore, the acoustic signal processing device 100a according to this modification can produce in a virtual space a personal space corresponding to the listener L and a sense of distance between the listener L and others that cannot be expressed by physical distance alone, by executing a second processing means that has a different predetermined value for each other person that is an object.

[0362] (Modification 2 of the embodiment) Modification 2 of the embodiment will be described below. The following description will focus on the differences from the embodiment, and the description of the commonalities will be omitted or simplified.

[0363] [Configuration] First, a description will be given of the configuration of an acoustic signal processing device 100b according to Modification 2 of the present embodiment. Fig. 12 is a block diagram showing the functional configuration of an acoustic signal processing device 100b according to this modification.

[0364] The acoustic signal processing device 100b according to this modification is capable of processing a plurality of pieces of sound data. The acoustic signal processing device 100b has the same configuration as the acoustic signal processing device 100 according to the embodiment, except that it includes a first processing unit 140b instead of the first processing unit 140.

[0365] 13 is a diagram showing an object and a plurality of sounds according to this modification. In this modification, the object in the virtual space is an ambulance A. The plurality of sounds includes three sounds, more specifically, two first sounds and one second sound.

[0366] The two first sounds and one second sound are sounds originating from the object, ambulance A. For the purpose of identification, the two first sounds are referred to as first sound A2 and first sound B2.

[0367] The first sound A2 is an aerodynamic sound (first aerodynamic sound) that occurs as the object (ambulance A) moves, i.e., wind noise. The first sound B2 is an aerodynamic sound that occurs when wind W emitted from the object (ambulance A) reaches the ears of the listener L, i.e., second aerodynamic sound. In other words, the wind W emitted from the object includes wind that is generated by the movement of an object such as a moving body (ambulance A), as shown in FIG.

[0368] The second sound is a sound different from the two first sounds, and more specifically, is a siren sound emitted from ambulance A. In this way, ambulance A is an object that generates multiple sounds including wind noise (first aerodynamic sound), second aerodynamic sound, and siren sound.

[0369] In this modification, the object information acquired by the first acquisition unit 110 includes first position information, first sound data indicating a first sound, first identification information, second sound data indicating a second sound, and second identification information indicating a processing means for the second sound data. The first sound data includes first sound data A2 indicating a first sound A2 and first sound data B2 indicating a first sound B2. The first identification information includes first identification information A2 indicating a processing means for the first sound data A2 and first identification information B2 indicating a processing means for the first sound data B2.

[0370] Furthermore, in the object information according to this modification, the position of the object (ambulance A) indicated by the first position information is associated with first sound data B2 indicating a first sound B2 (second aerodynamic sound) that is caused by the object and generated at a position related to the position of the listener L. Furthermore, although in real space the second aerodynamic sound is generated in the ear of the listener L, here the position of ambulance A in virtual space is treated as if it were the position of the sound source.

[0371] In other words, ambulance A is an object associated with a plurality of sound data including first sound data and second sound data (here, two sound data).

[0372] In this modified example, the determination unit 130 determines a processing means for processing the first sound data A2 based on the first identification information A2, determines a processing means for processing the first sound data B2 based on the first identification information B2, and determines a processing means for processing the second sound data based on the second identification information.

[0373] In this modified example, the first processing unit 140b processes the first sound data A2 using a processing means determined based on the first identification information A2, processes the first sound data B2 using a processing means determined based on the first identification information B2, and processes the second sound data using a processing means determined based on the second identification information.

[0374] Also, as in the embodiment, the second processing means is processing means that processes the first sound data so that the volume attenuates at the xth power (x≠1) of the distance D. And, because the first sound A2 is the first aerodynamic sound, the first identification information A2 indicates that the processing means for the first sound data A2 is the second processing means, and that x is α, α is a real number, and α satisfies (Equation 5).

[0375] α>1... (Formula 5)

[0376] Furthermore, since the first sound B2 is the second aerodynamic sound, the first identification information B2 indicates that the processing means for the first sound data B2 is the second processing means, and that x is β, β is a real number, and β satisfies (Equation 6).

[0377] β>2...(Formula 6)

[0378] Note that α and β satisfy (Equation 7).

[0379] α<β (Equation 7)

[0380] For example, α is 2 and β is 2.5.

[0381] The first sound B2 will now be described further. As described above, in the object information, the first sound data B2 indicating the first sound B2 (second aerodynamic sound) is associated with the position of the object (ambulance A). Therefore, by processing the first sound data B2 so that the volume of the first sound B2 attenuates as the distance D increases, the first processing unit 140b can make the first sound B2 (second aerodynamic sound) in the virtual space heard by the listener L resemble the first sound B2 (second aerodynamic sound) in the real space heard by the listener L.

[0382] Hereinafter, a third operational example of the acoustic signal processing method performed by the acoustic signal processing device 100b will be described.

[0383] [Operation Example 3] FIG. 14 is a flowchart of operation example 3 of the acoustic signal processing device 100b according to this modification.

[0384] 14, first, the first accepting unit 160 accepts an operation to specify the value of α, which is an example of x, indicated in the first identification information A2, and an operation to specify the value of β, which is an example of x, indicated in the first identification information B2 (S10b). This step S10b corresponds to the first accepting step.

[0385] As a result, x indicated in the first identification information A2 included in the object information stored in the storage unit 170 becomes α, and x indicated in the first identification information B2 becomes β.

[0386] Furthermore, the first acquisition unit 110 acquires object information including the first location information, the first sound data A2, the first identification information A2, the first sound data B2, the first identification information B2, the second sound data, and the second identification information, and the second location information (S20b). This step S20b corresponds to the first acquisition step.

[0387] Next, the first calculation unit 120 calculates the distance D between the object (ambulance A) and the listener L based on the first location information contained in the object information acquired by the first acquisition unit 110 and the second location information acquired by the first acquisition unit 110 (S30).

[0388] Next, the determination unit 130 determines processing means for processing the first sound data A2, the first sound data B2, and the second sound data based on the first identification information A2, the first identification information B2, and the second identification information included in the object information acquired by the first acquisition unit 110 (S40b). This step S40b corresponds to the determination step.

[0389] Next, the first processing unit 140b processes the first sound data A2, the first sound data B2, and the second sound data (S50b) using the processing means determined by the determination unit 130. This step S50b corresponds to the first processing step.

[0390] The first output unit 150 outputs the first sound data A2, the first sound data B2, and the second sound data processed by the first processing unit 140b (S60b). This step S60b corresponds to the first output step.

[0391] In this modified example, the means for processing the volume of the first sound can be changed in accordance with the first identification information, and the means for processing the volume of the second sound can be changed in accordance with the second identification information. Therefore, the first sound and the second sound in the virtual space heard by the listener L are similar to the first sound and the second sound in the real space heard by the listener L, respectively. Furthermore, because the first sound A2 is the first aerodynamic sound and the first sound B2 is the second aerodynamic sound, the second processing means is executed with different values ​​of x for the first sound data A2 and the first sound data B2. Therefore, the first sound A2 (first aerodynamic sound) and the first sound B2 (second aerodynamic sound) in the virtual space heard by the listener L are similar to the first sound A2 (first aerodynamic sound) and the first sound B2 (second aerodynamic sound), respectively, in the real space heard by the listener L. As a result, the listener L is less likely to feel uncomfortable, and can get a stronger sense of realism.

[0392] In the above description of this modified example, the object is ambulance A, but the object is not limited to this.

[0393] If the object is an object that generates multiple sounds, that is, if the object is an object that is associated with multiple sound data including first sound data and second sound data (two sound data in the above example), the same processing as in this modified example is performed. Below, first and second examples of objects that are subjected to such similar processing are described.

[0394] <First Example> In the first example, the object is an electric fan F. Fig. 15 is a diagram showing a case in which the object according to this modified example is an electric fan F. In this case, the first sound A2 is a first aerodynamic sound that is generated as the object, the electric fan F, moves, more specifically, as the blades of the electric fan F rotate, that is, a wind noise. The first sound B2 is a second aerodynamic sound that is generated when the wind W emitted from the object (electric fan F) reaches the ears of the listener L when the first sound reaches the ears of the listener L. The second sound is the motor sound of the electric fan F.

[0395] In this first example as well, the first sound and the second sound in the virtual space heard by the listener L are similar to the first sound and the second sound in the real space heard by the listener L. This makes it less likely that the listener L will feel uncomfortable, and allows the listener L to experience a greater sense of realism.

[0396] In addition, in the first example of the second modified example, a second processing means may be used that processes the first sound data so that the volume varies depending on whether the distance D is within a predetermined threshold value, as shown in the first and second examples of the first modified example of the embodiment.

[0397] In the first and second examples of Modification 1 of the embodiment, the second processing means is processing means that processes the first sound data so that the volume becomes a predetermined value when the distance D is within a predetermined threshold, but is not limited to this, and in the first example of Modification 2, the second processing means may be the following processing means: The second processing means in the first example of Modification 2 may be processing means that processes the first sound data so that the value of x when the distance D is within a predetermined threshold is larger than the value of x when the distance D is outside the predetermined threshold.

[0398] <Second Example> In the second example, the object is a zombie Z that appears in content displayed on the display unit 30. FIG. 16 is a diagram showing a case in which the object according to this modified example is a zombie Z. The zombie Z, which is the object, generates one first sound and one second sound. For example, the first sound is a groan emitted by the zombie Z, and the second sound is the footsteps of the zombie Z. In this case, the second processing means is processing means that processes the first sound data so that the volume attenuates as the xth power of the distance D (x≠1), where x is preferably a value greater than 1.

[0399] In step S50b shown in FIG. 14, for example, the first processing unit 140b processes the first sound data with the second processing means, and processes the second sound data with the first processing means.

[0400] In real space, the volume of a human voice (moan) attenuates in inverse proportion to the distance D between the living creature and the listener L. In other words, if the first sound data indicating the first sound, which is the moan of zombie Z, is processed by the first processing means, the listener L will hear the same voice (moan) as in real space. However, by deliberately processing the first sound data by the second processing means, that is, by processing it so that a moan different from that in real space is heard, the listener L can experience the eeriness of zombie Z, a fictional creature.

[0401] In the second example of this modified example 2, as in the first example of modified example 2, the second processing means may be processing means that processes the first sound data so that, for example, the value of x when the distance D is within a predetermined threshold is larger than the value of x when the distance D is outside the predetermined threshold.

[0402] (Modification 3 of the embodiment) Modification 3 of the embodiment will be described below. The following description will focus on the differences from the embodiment, and the description of the commonalities will be omitted or simplified.

[0403] [Configuration] First, a description will be given of the configuration of the information generating device 40 and the acoustic signal processing device 200 according to Modification 3 of the present embodiment. Fig. 17 is a block diagram showing the functional configuration of the information generating device 40 and the acoustic signal processing device 200 according to this modification.

[0404] The information generating device 40 and the acoustic signal processing device 200 according to this modification can prevent the occurrence of the problem described in (The knowledge that forms the basis of this disclosure) in which the listener L ends up hearing the second aerodynamic sound that causes discomfort. Fig. 18 is a diagram showing a fan F, which is an object according to this modification, and the listener L.

[0405] The information generating device 40 is a device that generates first object audio information and outputs it to the acoustic signal processing device 200. The acoustic signal processing device 200 is a device that acquires the output first object audio information and outputs sound data to the headphones 20 based on the acquired first object audio information.

[0406] First, the information generating device 40 shown in FIG. 17 will be described.

[0407] The information generating device 40 includes a receiving unit 41 , a second acquiring unit 42 , a first generating unit 43 , an output unit 44 , and a storage unit 45 .

[0408] The reception unit 41 receives operations from a user (e.g., a creator of content executed in a virtual space) of the information generation device 40. Specifically, the reception unit 41 is realized by a hardware button, but may also be realized by a touch panel or the like.

[0409] The second acquisition unit 42 acquires first sound data indicating a first sound generated at a position related to the position of the listener L in the virtual space, and first position information indicating the position of an object in the virtual space. In this modification, the object is an object that emits wind W, and is an electric fan F as shown in FIG. 18 . Furthermore, in the virtual space, the listener L is in a position where he or she is exposed to the wind W emitted from the electric fan F. The first sound is a sound generated at a position related to the position of the listener L, and in this case, the position related to the position of the listener L is the position of the ear of the listener L. In other words, the first sound is a sound caused by an object and generated at a position related to the position of the listener L (i.e., the position of the ear of the listener L). More specifically, the first sound according to this modification is aerodynamic sound (second aerodynamic sound) generated when the wind W emitted from the object, the electric fan F, reaches the ear of the listener L.

[0410] In this modification, the receiving unit 41 receives an operation indicating the first sound data and the first location information from the user. That is, the first sound data and the first location information are input by the user operating the receiving unit 41, and the second acquiring unit 42 acquires the input first sound data and first location information.

[0411] The first generation unit 43 generates first object audio information including information about an object that causes the first sound to be played at a position related to the position of the listener L due to the object and the first position information from the first sound data and the first position information acquired by the second acquisition unit 42.

[0412] The information about the object indicates that the object is an electric fan F, that the object itself is responsible for reproducing the first sound, and so on.

[0413] Furthermore, in the first object audio information according to this modification, the position of the object (electric fan F) indicated by the first position information is associated with first sound data indicating a first sound (second aerodynamic sound) that is caused by the object and generated at a position related to the position of the listener L. Furthermore, although in real space the second aerodynamic sound is generated in the ear of the listener L, here the position of the electric fan F in virtual space is treated as if it were the position of the sound source.

[0414] The first generator 43 may generate first object audio information including directivity information and unit distance information.

[0415] The directivity information is information that indicates characteristics according to the direction of the wind W emitted from the electric fan F. The directivity information is, for example, a database in which the wind speed for each direction in which the wind W is emitted or the attenuation rate of the volume of the second aerodynamic sound is stored in the SOFA (Spatially Oriented Format for Acoustics) format.

[0416] Furthermore, the unit distance information is information that includes a unit distance that is a reference distance, and aerodynamic sound data that indicates aerodynamic sound at a position that is the unit distance away from the position of the object indicated by the first position information. The aerodynamic sound data indicated in this unit distance information is data that indicates aerodynamic sound (second aerodynamic sound) at a position that is the unit distance away from the position of the object indicated in the directivity information, in the front direction in which the object radiates wind W.

[0417] Here, the directivity information and unit distance information will be described with reference to FIG.

[0418] Fig. 19 is a diagram for explaining directivity information and unit distance information according to this modification. The forward direction of the wind W emitted from the object (electric fan F) is defined as direction Df. Here, the direction that forms an angle θ1 with direction Df is defined as direction D31, and the direction that forms an angle θ2 with direction Df is defined as direction D32. Fig. 19 shows the unit distance, and a circle is illustrated at a position unit distance away from the object (electric fan F). In other words, the radius of the circle is the unit distance.

[0419] The wind speed of the wind W at a position unit distance away from the object (electric fan F) in the direction DF is defined as wsF. Then, the wind speed of the wind W at a position unit distance away from the object (electric fan F) in the direction D31 is defined as ws1, and the wind speed of the wind W at a position unit distance away from the object (electric fan F) in the direction D32 is defined as ws2. Then, ws1 and ws2 satisfy (Equation 8) and (Equation 9).

[0420] ws1 = wsF × C1... (Formula 8)

[0421] ws2 = wsF × C2... (Formula 9)

[0422] The directivity information is a database that indicates values ​​such as C1 and C2 (C1 and C2 are constants) for each angle such as θ1 and θ2. Note that C1 is described as the value at angle θ1 indicated by the directivity information, and C2 is described as the value at angle θ2 indicated by the directivity information.

[0423] In the above example, the directivity regarding wind speed was described, but the directivity regarding the volume of the sound caused by the wind W (second aerodynamic sound) may also be described in a similar manner.

[0424] The first generator 43 also generates first object audio information that further includes flag information. The flag information is information indicating whether or not to perform processing to convolve sound data (here, aerodynamic sound data) representing sound generated from the object (electric fan F) with a head-related transfer function that depends on the direction from which the sound arrives during sound playback.

[0425] The directivity information, the unit distance information, and the flag information may be stored in advance in, for example, the storage unit 45. The first generation unit 43 may acquire the directivity information, the unit distance information, and the flag information from the storage unit 45, or may acquire them by operating the reception unit 41, similarly to the first sound data and the first position information.

[0426] The output unit 44 outputs the first sound data and first position information acquired by the second acquisition unit 42 and the first object audio information generated by the first generation unit 43 to the acoustic signal processing device 200 .

[0427] The storage unit 45 is a storage device that stores computer programs executed by the reception unit 41, the second acquisition unit 42, the first generation unit 43, and the output unit 44, etc.

[0428] Next, the acoustic signal processing device 200 shown in FIG. 17 will be described.

[0429] As shown in FIG. 17, the acoustic signal processing device 200 includes a third acquisition unit 210 , a second calculation unit 220 , a second processing unit 240 , a second output unit 250 , and a storage unit 270 .

[0430] The third acquisition unit 210 acquires the first object audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and second position information indicating the position of the listener L of the first sound. The third acquisition unit 210 acquires the second position information from the headphones 20 (more specifically, the head sensor unit 21), but is not limited to this.

[0431] The second calculation unit 220 calculates the distance D between the object (fan F) and the listener L based on the first position information included in the first object audio information acquired by the third acquisition unit 210 and the acquired second position information. The second calculation unit 220 calculates the distance D in the same manner as the first calculation unit 120 according to the embodiment.

[0432] Furthermore, the second calculation unit 220 calculates the point-to-point direction connecting the object (fan F) and the listener L based on the first position information included in the first object audio information acquired by the third acquisition unit 210 and the acquired second position information.

[0433] The second processing unit 240 processes the first sound data so that the volume of the first sound attenuates as the distance D calculated by the second calculation unit 220 increases. For example, the second processing unit 240 may process the first sound data using the second processing means shown in Modification 2 of the embodiment. As described above, since the first sound is the second aerodynamic sound, the first sound data may be processed using β as the value of x. In this case, β is 2.5, for example.

[0434] Furthermore, in the first object audio information according to this modification, first sound data indicating the first sound (second aerodynamic sound) is associated with the position of the object (electric fan F). For this reason, the second processing unit 240 processes the first sound data B2 so that the volume of the first sound B2 attenuates as the distance D increases, thereby making it possible for the first sound (second aerodynamic sound) in the virtual space heard by the listener L to resemble the first sound (second aerodynamic sound) in the real space heard by the listener L.

[0435] Furthermore, the second processing unit 240 may perform the following processing when the calculated distance D is greater than the unit distance indicated by the unit distance information included in the first object audio information acquired by the third acquisition unit 210. In other words, in this case, the second processing unit 240 processes the first sound data so as to attenuate the volume of the first sound in accordance with the calculated distance D and the unit distance. This processing will be described with reference to FIG. 20 .

[0436] FIG. 20 is a diagram for explaining the processing of the second processing unit 240 according to this modified example. FIG. 20 shows the positional relationship between an object (electric fan F) and a listener L. The listener L is located in the front direction of the wind W emitted from the object (electric fan F). As explained above, the unit distance information indicates aerodynamic sound data that indicates aerodynamic sound at a position that is a unit distance away from the position of the object in the front direction. In the example shown in FIG. 20 , the first sound data corresponds to that aerodynamic sound data. When the calculated distance D is greater than the unit distance, the second processing unit 240 processes the aerodynamic sound data that indicates aerodynamic sound at a position that is a unit distance away from the position of the object so that the volume of the first sound (second aerodynamic sound) decreases as the distance D increases.

[0437] Furthermore, the second processing unit 240 may process the first sound data so as to control the volume of the first sound based on the angle between the front direction of the wind W emitted from the object (electric fan F) and the point-to-point direction calculated by the second calculation unit 220 and the characteristics indicated by the directivity information. This processing will be described with reference to FIG. 21 .

[0438] Fig. 21 is a diagram illustrating another process performed by the second processing unit 240 according to this modified example. Fig. 21 shows the positional relationship between an object (electric fan F) and a listener L. The listener L is located in a direction D31 from the object (electric fan F). In this case, the angle between the front direction and the direction between the two points is θ1. If the listener L is located in the position shown in Fig. 21 (the position of the listener L indicated by the solid line), the wind speed of the wind W experienced by the listener L will be lower than if the listener L is located in a position in the front direction (the position of the listener L indicated by the dashed line).

[0439] 21 is equal to the wind speed experienced by the listener L located in the front direction multiplied by the above C1. Because the volume of the first sound (second aerodynamic sound) heard by the listener L changes depending on the wind speed experienced by the listener L, the second processing unit 240 may process the first sound data (here, the aerodynamic sound data indicated in the unit distance information) depending on the wind speed.

[0440] It is preferable that the second processing unit 240 performs both the processing described in FIG. 20 and the processing described in FIG. 21, and outputs one piece of processed first sound data (aerodynamic sound data) to the second output unit 250.

[0441] The second output unit 250 outputs the first sound data (aerodynamic sound data) processed by the second processing unit 240. Here, the second output unit 250 outputs the first sound data to the headphones 20. This allows the headphones 20 to play back the first sound indicated by the output first sound data.

[0442] The storage unit 270 is a storage device that stores computer programs executed by the third acquisition unit 210, the second calculation unit 220, the second processing unit 240, and the second output unit 250, and the like.

[0443] Below, a fourth operational example of the information generating method performed by the information generating device 40 and a fifth operational example of the acoustic signal processing method performed by the acoustic signal processing device 200 will be described.

[0444] [Operation Example 4] FIG. 22 is a flowchart of operation example 4 of the information generating device 40 according to this modification.

[0445] 22 , first, the receiving unit 41 receives an operation indicating the first sound data and the first location information from the user (S110). That is, the first sound data and the first location information are input by the user operating the receiving unit 41.

[0446] Next, the second acquisition unit 42 acquires first sound data indicating a first sound occurring at a position related to the position of the listener L in the virtual space, and first position information indicating the position of an object in the virtual space (S120). Here, the second acquisition unit 42 acquires the first sound data and first position information input in step S110. This step S120 corresponds to the second acquisition step.

[0447] Next, the first generating unit 43 generates first object audio information including information about the object, the first position information, unit distance information, and directivity information from the first sound data and the first position information acquired by the second acquiring unit 42 (S130). The generated first object audio information may also include flag information. Step S130 corresponds to the first generating step.

[0448] Furthermore, the output unit 44 outputs the first sound data and first position information acquired by the second acquisition unit 42 and the first object audio information generated by the first generation unit 43 to the acoustic signal processing device 200 (S140).

[0449] [Operation Example 5] FIG. 23 is a flowchart of operation example 5 of the acoustic signal processing device 200 according to this modification.

[0450] 23 , first, the third acquisition unit 210 acquires the first object audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and the second position information output from the headphones 20 (S210). This step S210 corresponds to a third acquisition step.

[0451] Next, the second calculation unit 220 calculates the distance D between the object (electric fan F) and the listener L based on the first position information included in the first object audio information acquired by the third acquisition unit 210 and the acquired second position information, and also calculates the point-to-point direction connecting the object (electric fan F) and the listener L based on the first position information included in the first object audio information acquired by the third acquisition unit 210 and the acquired second position information (S220). This step S220 corresponds to the second calculation step.

[0452] Next, the second processing unit 240 processes the first sound data so as to control the volume of the first sound based on the angle between the above-mentioned front direction and the direction between the two points calculated by the second calculation unit 220 and the characteristics indicated by the directivity information, and so as to attenuate the volume of the first sound in accordance with the calculated distance D and the unit distance when the calculated distance D is greater than the unit distance indicated by the unit distance information included in the first object audio information acquired by the third acquisition unit 210 (S230).

[0453] Furthermore, if the first object audio information includes flag information, the second processing unit 240 determines, in accordance with the flag information, whether or not to perform processing to convolve a head-related transfer function (HRTF) that depends on the direction from which the sound arrives onto the first sound signal based on the first sound data. Here, as an example, the second processing unit 240 performs processing to convolve a head-related transfer function (HRTF) onto the first sound signal based on the first sound data in accordance with the flag information (S240). These steps S230 and S240 correspond to the second processing step.

[0454] Then, the second output unit 250 outputs the first sound data (first sound signal) processed by the second processing unit 240 (S250). This step S250 corresponds to a second output step.

[0455] The information generation method according to this modification can generate first object audio information in which the position of an object is associated with first sound data indicating a first sound (second aerodynamic sound) that is caused by the object and occurs at a position related to the position of the listener L. Furthermore, the acoustic signal processing method according to this modification processes the first sound data so that the volume of the first sound (second aerodynamic sound) attenuates as the distance D between the object and the listener L increases, so that the first sound (second aerodynamic sound) in the virtual space heard by the listener L resembles the first sound (second aerodynamic sound) in the real space heard by the listener L. In other words, the listener L does not hear the first sound (second aerodynamic sound) that causes an unpleasant feeling, and the listener L can obtain a sense of presence.

[0456] (Fourth Modification of the Embodiment) The following describes a fourth modification of the embodiment, focusing on differences from the third modification of the embodiment, and omitting or simplifying the description of commonalities.

[0457] [Configuration] First, a description will be given of the configuration of the information generating device 40 and the acoustic signal processing device 200c according to Modification 4 of the present embodiment. Fig. 24 is a block diagram showing the functional configuration of the information generating device 40 and the acoustic signal processing device 200c according to this modification.

[0458] This modification uses the information generating device 40 according to modification 3. The acoustic signal processing device 200c has the same configuration as the acoustic signal processing device 200 according to modification 3, except that it does not include the second calculation unit 220 and includes a second processing unit 240c instead of the second processing unit 240.

[0459] In this modification, the acoustic signal processing device 200c is a device that handles a first sound caused by an object and generated at a position related to the position of the listener L, and a second sound caused by the object. The object in this modification is the ambulance A, as in the second modification.

[0460] The first sound is an aerodynamic sound, that is, a second aerodynamic sound, which is generated when wind W emitted from an object (ambulance A) reaches the ears of a listener L. The second sound is a siren sound emitted from ambulance A.

[0461] As shown in FIG. 24, the acoustic signal processing device 200 c includes a third acquisition unit 210 , a second processing unit 240 c , a second output unit 250 , and a storage unit 270 .

[0462] The third acquisition unit 210 according to this modification acquires first object audio information generated by the information generating device 40, first sound data acquired by the information generating device 40, and second object audio information. The second object audio information is information in which first position information indicating the position of an object (ambulance A) in a virtual space is associated with second sound data indicating a second sound caused by the object (ambulance A). Note that the second object audio information is data in which the second sound data indicating the second sound is associated with first position information indicating the position at which the second sound is generated, and therefore corresponds to object audio information in conventional technologies such as the technology disclosed in Patent Document 1.

[0463] The second object audio information may be generated by the information generating device 40 and output to the acoustic signal processing device 200c. The third acquiring unit 210 acquires the output second object audio information.

[0464] The second processing unit 240c processes the first sound data acquired by the third acquisition unit 210 and the second sound data included in the second object audio information acquired by the third acquisition unit 210 as follows.

[0465] First, the processing of the second sound data will be described.

[0466] The second processing unit 240c performs a process of convolving a head-related transfer function that depends on the direction from which the sound arrives with the second sound signal based on the second sound data indicated by the second object audio information acquired by the third acquisition unit 210.

[0467] Next, the processing of the first sound data will be described.

[0468] As one example, the second processing unit 240c does not perform processing of convolving a head-related transfer function that depends on the direction from which sound arrives, on the first sound signal based on the first sound data acquired by the third acquisition unit 210. As another example, the second processing unit 240c performs processing that depends on the direction from which wind W arrives from the ambulance A toward the listener L, on the first sound signal based on the first sound data acquired by the third acquisition unit 210. This other example of processing will be described with reference to FIG. 25 .

[0469] FIG. 25 is a diagram for explaining the processing of the first sound data according to this modified example.

[0470] As shown in Fig. 25, the speed and volume of the wind W arriving at the right and left ears of the listener L differ depending on the positional relationship between the object (ambulance A) and the listener L. In Fig. 25, the wind W arrives faster and at a larger volume at the left ear, and slower and at a smaller volume at the right ear.

[0471] For this reason, the second processing unit 240c may perform the following processing on the first sound data as processing dependent on the direction in which the wind W is coming from the ambulance A toward the listener L. In other words, the first sound data may be processed so that the first sound (second aerodynamic sound) heard by the listener L becomes a sound that simulates the time difference and wind volume ratio for the first sound to reach both ears of the listener L. This allows the listener L to perceive the direction of the source of the wind W.

[0472] The first object audio information includes processing information indicating whether the second processing unit 240c should not convolve the first sound signal with a head-related transfer function, or should perform processing on the first sound signal that depends on the direction in which the wind W arrives from the ambulance A toward the listener L. The second processing unit 240c processes the first sound signal in accordance with the processing information included in the first object audio information.

[0473] The second output unit 250 according to this modification outputs the second sound signal processed by the second processing unit 240c. If the second processing unit 240c has not processed the first sound signal, the second output unit 250 outputs the unprocessed first sound signal. If the second processing unit 240c has processed the first sound signal, the second output unit 250 outputs the processed first sound signal.

[0474] In addition, the memory unit 270 in this modified example stores the head-related transfer function used by the second processing unit 240c and information necessary for processing that depends on the direction in which the wind W arrives from the ambulance A toward the listener L.

[0475] Hereinafter, a fifth and sixth operation examples of the acoustic signal processing method performed by the acoustic signal processing device 200c will be described.

[0476] 26 is a flowchart of operation example 6 of the acoustic signal processing device 200c according to this modification. In this operation example, the second processing unit 240c does not process the first sound signal.

[0477] 26, first, the third acquisition unit 210 acquires the first object audio information generated by the information generator 40, the first sound data acquired by the information generator 40, and the second object audio information in which the first position information and the second sound data are associated with each other (S310). This step S310 corresponds to the third acquisition step.

[0478] The second processing unit 240c does not perform the process of convolving a head-related transfer function that depends on the direction from which the sound arrives with the first sound signal based on the first sound data acquired by the third acquisition unit 210 (S320).

[0479] The second processing unit 240c performs processing to convolve a head-related transfer function that depends on the direction from which the sound arrives with the second sound signal based on the second sound data indicated by the second object audio information acquired by the third acquisition unit 210 (S330). These steps S320 and S330 correspond to the second processing step.

[0480] The second output unit 250 outputs the first sound signal that has not been processed by the second processing unit 240c and the second sound signal that has been processed by the second processing unit 240c (S340). This step S340 corresponds to a second output step.

[0481] 27 is a flowchart of operation example 7 of the acoustic signal processing device 200c according to this modification. In this operation example, the second processing unit 240c processes the first sound signal.

[0482] As shown in FIG. 27, first, step S310 is performed.

[0483] The second processing unit 240c processes the first sound signal based on the first sound data acquired by the third acquisition unit 210 in a manner that depends on the direction in which the wind W is coming from the ambulance A to the listener L (S320c).

[0484] Next, step S330 is performed.

[0485] Then, the second output unit 250 outputs the first sound signal processed by the second processing unit 240c and the second sound signal processed by the second processing unit 240c (S340c).

[0486] In this modification, the second sound in the virtual space heard by the listener L is similar to the second sound in the real space heard by the listener L because the head-related transfer function is convolved with the second sound signal based on the second sound data. This makes it less likely that the listener L will feel uncomfortable, and allows the listener L to experience a greater sense of realism.

[0487] Furthermore, in Operation Example 7, the first sound in the virtual space heard by the listener L is similar to the first sound in the real space heard by the listener L because the first sound signal based on the first sound data is processed in a manner that depends on the direction from which the wind W is coming. This makes it less likely for the listener L to feel uncomfortable, and allows the listener L to experience a greater sense of realism.

[0488] (Fifth Modification of the Embodiment) The following describes a fifth modification of the embodiment, focusing on differences from the fourth modification of the embodiment, and omitting or simplifying the description of commonalities.

[0489] [Configuration] First, a description will be given of the configuration of the information generating device 40 and the acoustic signal processing device 200d according to Modification 5 of the present embodiment. Fig. 28 is a block diagram showing the functional configuration of the information generating device 40 and the acoustic signal processing device 200d according to this modification.

[0490] In this modification, the information generating device 40 according to modification 4 is used. The acoustic signal processing device 200d has the same configuration as the acoustic signal processing device 200c according to modification 4, except that it includes a second processing unit 240d instead of the second processing unit 240c.

[0491] Variation 4 deals with a first sound (second aerodynamic sound) that is caused by an object, ambulance A, and occurs at a position related to the position of listener L, and a second sound (siren sound) that is caused by the object, ambulance A. Variation 5 deals with a first sound (second aerodynamic sound) that is caused by an object, ambulance A, and occurs at a position related to the position of listener L, and a third sound that is caused by another object in the virtual space that is different from the object.

[0492] The other object, like the one object, is not particularly limited as long as it is included in the content (here, video as an example) displayed on the display unit 30 that displays the content executed in the virtual space. The one object and the other object are provided in the same virtual space. For simplicity, hereinafter, the one object may be simply referred to as the object, and the other object may be simply referred to as the other object.

[0493] The third sound is a sound generated at the position of another object in the virtual space, and is different from the first aerodynamic sound and the second aerodynamic sound.

[0494] In this manner, in this modified example, a plurality of objects including the first sound and the third sound, or more specifically, one object and another object, are handled.

[0495] The third acquisition unit 210 according to this modification acquires the first object audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and third object audio information. The third object audio information is information in which third position information indicating the position of another object in the virtual space is associated with third sound data indicating a third sound generated at the position of the other object. Note that the third object audio information is data in which the third sound data indicating the third sound is associated with the third position information indicating the position at which the third sound is generated, and therefore corresponds to object audio information in conventional technologies such as the technology disclosed in Patent Document 1.

[0496] The third object audio information may be generated by the information generating device 40 and output to the acoustic signal processing device 200d. The third acquiring unit 210 acquires the output third object audio information.

[0497] The second processing unit 240d processes the first sound data acquired by the third acquisition unit 210 and the third sound data included in the third object audio information acquired by the third acquisition unit 210 as follows.

[0498] The second processing unit 240d performs processing on the first sound signal based on the first sound data that depends on the direction in which the wind W is coming from the ambulance A toward the listener L. The second processing unit 240d also performs processing on the third sound signal based on the third sound data indicated by the acquired third object audio information, convolving a head-related transfer function that depends on the direction in which the sound is coming. That is, in this modification, the second processing unit 240d performs the same processing on the first sound signal as the processing described with reference to Fig. 25 . The second processing unit 240d also performs the same processing on the third sound signal as the second sound signal according to Modification 4.

[0499] The second output section 250 according to this modification outputs the first sound signal processed by the second processing section 240d and the third sound signal processed by the second processing section 240d.

[0500] An eighth operational example of the acoustic signal processing method performed by the acoustic signal processing device 200d will now be described.

[0501] [Operation Example 8] FIG. 29 is a flowchart of operation example 8 of the acoustic signal processing device 200d according to this modification.

[0502] 29 , first, the third acquisition unit 210 acquires the first object audio information generated by the information generator 40, the first sound data acquired by the information generator 40, and third object audio information in which third position information indicating the position of another object and third sound data indicating a third sound generated at the position of the another object are associated (S310d). This step S310d corresponds to the third acquisition step.

[0503] The second processing unit 240d processes the first sound signal based on the first sound data acquired by the third acquisition unit 210 in a manner that depends on the direction in which the wind W is coming from the ambulance A to the listener L (S320c).

[0504] The second processing unit 240d performs processing to convolve a head-related transfer function that depends on the direction from which the sound arrives with the third sound signal based on the third sound data indicated by the third object audio information acquired by the third acquisition unit 210 (S330d). These steps S320c and S330d correspond to the second processing step.

[0505] Then, the second output unit 250 outputs the first sound signal processed by the second processing unit 240d and the third sound signal processed by the second processing unit 240d (S340d). This step S340d corresponds to a second output step.

[0506] In this modification, when a plurality of objects including an object and other objects are provided in a virtual space, the first sound and the third sound in the virtual space heard by the listener L are similar to the first sound and the third sound in the real space heard by the listener L. As a result, the listener L is less likely to feel uncomfortable, and can obtain a greater sense of realism.

[0507] (Sixth Modification of the Embodiment) The sixth modification of the embodiment will be described below. The following description will focus on the differences from the third modification of the embodiment, and the description of the commonalities will be omitted or simplified.

[0508] [Configuration] First, a description will be given of the configuration of the information generating device 50 and the acoustic signal processing device 400 according to Modification 6 of the present embodiment. Fig. 30 is a block diagram showing the functional configuration of the information generating device 50 and the acoustic signal processing device 400 according to this modification.

[0509] As with Modification 3, the information generating device 50 and the acoustic signal processing device 400 according to this modification can suppress the occurrence of the problem described in (The Knowledge Forming the Basis of the Present Disclosure) that the listener L ends up hearing the second aerodynamic sound that causes discomfort. In this modification, as with Modification 3, the object is an electric fan F, but this is not limited to this and any object that can emit wind W can be used. Furthermore, the wind W emitted from an object also includes wind that is generated by the movement of an object such as a moving body (ambulance A), as shown in FIG. 13 .

[0510] The information generating device 50 is a device that generates fourth object audio information and outputs it to the acoustic signal processing device 400. The acoustic signal processing device 400 is a device that acquires the output fourth object audio information and outputs sound data to the headphones 20 based on the acquired fourth object audio information.

[0511] First, the information generating device 50 shown in FIG. 30 will be described.

[0512] The information generating device 50 includes a second receiving unit 51 , a fourth acquiring unit 52 , a second generating unit 53 , a third output unit 54 , and a storage unit 55 .

[0513] The second reception unit 51 receives an operation from a user (e.g., a creator of content executed in a virtual space) of the information generation device 50. Specifically, the second reception unit 51 is realized by a hardware button, but may also be realized by a touch panel or the like.

[0514] The fourth acquisition unit 52 acquires the generation position of a first wind blowing in the virtual space, a first wind direction of the first wind, and a first estimated wind speed.

[0515] In this modified example, the first wind blowing in the virtual space is wind W radiated from an electric fan F, which is an object. In other words, the position where the first wind is generated is the position where the electric fan F is placed. Note that in this modified example, it is sufficient that the first wind is blowing in the virtual space, and the electric fan F, which is the object radiating this first wind, does not need to be placed within the virtual space (more specifically, the virtual space where the listener L is located). In other words, it is sufficient that the electric fan F is placed outside the virtual space where the listener L is located, and the first wind, which is wind W radiated from this electric fan F, reaches the virtual space. Naturally, the electric fan F, which is the object radiating the first wind, may also be placed within the virtual space where the listener L is located.

[0516] The first wind direction is the wind direction of the first wind, which is the front direction of the wind W emitted from the object (electric fan F), for example, direction Df shown in Fig. 19. The first assumed wind speed may be any wind speed of the first wind, and in this case, it is the wind speed of the first wind at a position a unit distance away from the generation position in the direction of the first wind direction, which is a reference distance. In other words, the first assumed wind speed is, for example, wind speed wsF shown in Fig. 19.

[0517] As described above, in the virtual space, a first wind, which is wind W emitted from the object, electric fan F, is blowing, and the listener L hears aerodynamic sound (second aerodynamic sound) generated when this wind W (first wind) reaches the listener L's ears.

[0518] In this modification, the second receiving unit 51 receives an operation from the user indicating the generation position of the first wind, the first wind direction, and the first estimated wind speed. That is, the user operates the second receiving unit 51 to input the generation position of the first wind, the first wind direction, and the first estimated wind speed, and the fourth obtaining unit 52 obtains the input generation position of the first wind, the first wind direction, and the first estimated wind speed.

[0519] The second reception unit 51 may receive an operation from the user specifying that the unit distance is the second specified value. That is, the user operates the second reception unit 51 to set the unit distance to the second specified value.

[0520] The second generating unit 53 generates fourth object audio information in which the generation position, the first wind direction, and the first assumed wind speed acquired by the fourth acquiring unit 52 are associated with each other.

[0521] The second receiving unit 51 may receive, from the user, an operation to specify directivity information indicating characteristics according to the direction of the first wind. The directivity information is the same as the information described in FIG. 19 and other figures. When such an operation is received, the second generating unit 53 generates fourth object audio information in which the generation position, the first wind direction, and the first estimated wind speed are associated with the directivity information indicated by the operation received by the second receiving unit 51.

[0522] The storage unit 55 is a storage device that stores computer programs executed by the second receiving unit 51, the fourth acquiring unit 52, the second generating unit 53, and the third output unit 54. It is assumed that the storage unit 55 stores aerodynamic sound core information that includes a representative wind speed and aerodynamic sound data that indicates aerodynamic sound (second aerodynamic sound) that is generated when wind at the representative wind speed reaches the ears of the listener L in the virtual space. However, the aerodynamic sound core information does not necessarily have to be stored in the information generating device 50, and may be stored in a memory independent of the information generating device 50.

[0523] In this modified example, the second reception unit 51 receives an operation indicating the aerodynamic noise core information from the user. In other words, the aerodynamic noise core information is input by the user operating the second reception unit 51, and the input aerodynamic noise core information is stored in the storage unit 55.

[0524] The aerodynamic sound core information indicates a representative wind speed, which shows an example of one wind speed value, and aerodynamic sound data, which indicates the aerodynamic sound generated when wind at this representative wind speed reaches the ears of the listener L, and is a database used for information processing in the acoustic signal processing device 400, which will be described later. The aerodynamic sound data, which indicates the aerodynamic sound (second aerodynamic sound), indicates the volume of the second aerodynamic sound, etc. The aerodynamic sound core information may also include data, which indicates the distribution of frequency components of the aerodynamic sound (hereinafter, sometimes referred to as frequency data). For example, the frequency data is data, which indicates the frequency characteristics of the aerodynamic sound (second aerodynamic sound).

[0525] Such aerodynamic sound core information can also be used in the third processing unit 440, which will be described later, to determine the volume of the second aerodynamic sound heard by the listener L from the wind speed reaching the ear.

[0526] The aerodynamic sound core information may include multiple pairs of a representative wind speed and aerodynamic sound data at that representative wind speed. For example, as shown in Fig. 4C , the wind speed at each position where the wind speed differs due to a different distance from the source of the wind W is taken as the representative wind speed, and the aerodynamic sound core information may include multiple such pairs formed from the representative wind speed and the frequency data corresponding to that representative wind speed. Aerodynamic sound core information configured in this way can also be used in the third processing unit 440, described below, to determine the volume of the second aerodynamic sound heard by the listener L, using the wind speed reaching the ear as an index.

[0527] The third output unit 54 outputs the fourth object audio information generated by the second generation unit 53 and the aerodynamic sound core information stored in the storage unit 55 to the acoustic signal processing device 400 .

[0528] Next, an acoustic signal processing device 400 shown in FIG. 30 will be described.

[0529] As shown in FIG. 30 , the acoustic signal processing device 400 includes a fifth acquisition unit 410 , a third calculation unit 420 , a third processing unit 440 , a fourth output unit 450 , and a storage unit 470 .

[0530] The fifth acquisition unit 410 acquires the fourth object audio information and aerodynamic sound core information output by the information generating device 50, as well as second position information indicating the position of the listener L in the virtual space. The fifth acquisition unit 410 acquires the second position information from the headphones 20 (more specifically, the head sensor unit 21), but is not limited to this. In this example, the aerodynamic sound core information includes frequency data.

[0531] The third calculation unit 420 calculates the distance D between the generation position (i.e., the fan F) and the listener L, based on the generation position included in the fourth object audio information acquired by the fifth acquisition unit 410 and the acquired second position information. The third calculation unit 420 calculates the distance D in the same manner as the first calculation unit 120 according to the embodiment.

[0532] Furthermore, the third calculation unit 420 calculates the point-to-point direction connecting the object (fan F) and the listener L based on the generation position included in the fourth object audio information acquired by the fifth acquisition unit 410 and the acquired second position information.

[0533] The third processing unit 440 processes the aerodynamic sound data so that the volume of the aerodynamic sound decreases as the distance D calculated by the third calculation unit 420 increases.

[0534] Here, the third processing unit 440 may process the aerodynamic sound data so that the volume of the aerodynamic sound (second aerodynamic sound) attenuates at the γth power of the value obtained by dividing the representative wind speed by the ear arrival wind speed, which is the wind speed of the first wind when it reaches the ear of the listener L. In other words, the third processing unit 440 processes the aerodynamic sound data based on the distance D, the representative wind speed, and the ear arrival wind speed. Note that the ear arrival wind speed slows down as the distance D calculated by the third calculation unit 420 increases. Furthermore, the ear arrival wind speed is a value that attenuates at the zth power of the value obtained by dividing the distance D calculated by the third calculation unit 420 by the unit distance.

[0535] More specifically, the third processing unit 440 performs the following processing.

[0536] First, the angle between the first wind direction (for example, direction Df shown in FIG. 19 ), which is the front direction of the wind W, and the calculated direction between the two points is defined as θ. In this case, if the wind speed at the ear, which is the wind speed of the first wind when it reaches the ear of the listener L, is defined as Se1, Se1 satisfies (Equation 10).

[0537] Se1=first expected wind speed×value at angle θ indicated by directivity information×{(unit distance as reference distance / distance D)}^z (Equation 10)

[0538] For example, if θ is θ1, the value at angle θ (angle θ1) indicated by the directivity information is C1. Furthermore, the value obtained by dividing the representative wind speed by the ear arrival wind speed Se1 is set to R1. Furthermore, V3, which is the volume of the second aerodynamic sound heard by listener L, satisfies (Equation 11).

[0539] V3 = volume indicated by the aerodynamic noise data of the aerodynamic noise core information × {(1 / R1)}^γ ... (Equation 11)

[0540] Furthermore, z should satisfy (Equation 12).

[0541] z=1...(Formula 12)

[0542] Furthermore, γ and z should satisfy (Equation 13).

[0543] γ×z < 4... (Formula 13)

[0544] The method for calculating the volume V3 of the second aerodynamic sound heard by the listener L from the ear arrival wind speed Se1 is not limited to the above. For example, the aerodynamic sound core information may include multiple pairs of representative wind speeds and frequency data at those wind speeds, and the third processing unit 440 may detect frequency data for a representative wind speed close to Se1 and apply that frequency data to generate a second aerodynamic sound with a predetermined volume. In other words, rather than calculating volume V3 from Se1 using a formula, aerodynamic sound data may be detected using Se1 as an index, and the predetermined volume may be achieved by applying that data. The volume of the second aerodynamic sound is lower the smaller the wind speed, so the aerodynamic sound core information should be provided so that the resulting volume will be lower if frequency data for a low representative wind speed is applied. Because Se1 decreases as the distance D increases, sound V3 can be controlled to decrease as the distance D increases.

[0545] The fourth output unit 450 outputs the aerodynamic sound data processed by the third processing unit 440. Here, the fourth output unit 450 outputs the aerodynamic sound data to the headphones 20. This allows the headphones 20 to play back the second aerodynamic sound indicated by the output aerodynamic sound data.

[0546] The storage unit 470 is a storage device that stores computer programs executed by the fifth acquisition unit 410, the third calculation unit 420, the third processing unit 440, and the fourth output unit 450, etc.

[0547] If the aerodynamic noise core information acquired by the fifth acquisition unit 410 includes frequency data, the third processing unit 440 may perform the following processing.

[0548] The third processing unit 440 processes the aerodynamic sound data so that the distribution of frequency components of the aerodynamic sound (second aerodynamic sound) shifts toward lower frequencies as the distance D calculated by the third calculation unit 420 increases.

[0549] Here, the third processing unit 440 processes the aerodynamic sound data so as to shift the distribution of aerodynamic sound frequency components to a frequency that is the reciprocal multiple of the value (R1 above) obtained by dividing the representative wind speed by the wind speed at ear. In other words, the distribution of frequency components included in the aerodynamic sound core information (the distribution of frequency components before processing) is processed by the third processing unit 440, resulting in a distribution of frequency components that has been shifted to a frequency that is the reciprocal of R1.

[0550] In this way, the third processing unit 440 processes the aerodynamic sound data based on the distance D, the representative wind speed, and the ear-reaching wind speed. The ear-reaching wind speed decreases as the calculated distance D increases. The ear-reaching wind speed is a value that decays at the zth power of the value obtained by dividing the calculated distance D by the unit distance. In this case, z also satisfies the above (Equation 12).

[0551] Below, a ninth operational example of the information generating method performed by the information generating device 50, and tenth and eleventh operational examples of the acoustic signal processing method performed by the acoustic signal processing device 400 will be described.

[0552] [Operation Example 9] FIG. 31 is a flowchart of operation example 9 of the information generating device 50 according to this modification.

[0553] 31 , first, the second reception unit 51 receives from the user an operation specifying that the unit distance is a second specified value and an operation specifying directivity information indicating characteristics according to the direction of the first wind (S410). At this time, the second reception unit 51 may receive from the user an operation specifying the location where the first wind is generated, the first wind direction, and the first estimated wind speed. That is, the user operates the second reception unit 51 to input the unit distance, directivity information, the location where the first wind is generated, the first wind direction, and the first estimated wind speed. This step S410 corresponds to a second reception step.

[0554] Next, the fourth acquisition unit 52 acquires the generation location of the first wind, the first wind direction of the first wind, and a first estimated wind speed, which is the wind speed of the first wind at a position a unit distance away from the generation location in the direction of the first wind direction, which is a reference distance (S420). The fourth acquisition unit 52 may also acquire directivity information. Here, the fourth acquisition unit 52 acquires the generation location of the first wind, the first wind direction of the first wind, the first estimated wind speed, and the directivity information input in step S410. This step S420 corresponds to the fourth acquisition step.

[0555] Next, the second generating unit 53 generates fourth object audio information in which the generation position, the first wind direction, the first assumed wind speed, and the directivity information are associated with each other (S430). This step S430 corresponds to the second generating step.

[0556] Next, aerodynamic sound core information including a representative wind speed and aerodynamic sound data indicating aerodynamic sound generated when wind at the representative wind speed reaches the ears of the listener L (S440) is stored in the storage unit 55. This aerodynamic sound core information may include data indicating the distribution of frequency components of the aerodynamic sound. Step S440 corresponds to the storage step.

[0557] The third output unit 54 outputs the fourth object audio information generated by the second generation unit 53 and the aerodynamic sound core information stored in the storage unit 55 to the acoustic signal processing device 400 (S450). This step S450 corresponds to the third output step.

[0558] 32 is a flowchart of operation example 10 of the acoustic signal processing device 400 according to this modification. Operation example 10 is an example in which the third processing unit 440 controls the volume of aerodynamic sound.

[0559] 32 , first, the fifth acquisition unit 410 acquires the fourth object audio information and aerodynamic sound core information output by the information generating device 50, and the second position information indicating the position of the listener L in the virtual space (S510). This step S510 corresponds to the fifth acquisition step.

[0560] Next, the third calculation unit 420 calculates the distance D between the generation position included in the fourth object audio information acquired by the fifth acquisition unit 410 and the acquired second position information (S520). At this time, the third calculation unit 420 calculates the direction between two points connecting the object (electric fan F) and the listener L based on the generation position included in the fourth object audio information acquired by the fifth acquisition unit 410 and the acquired second position information. Step S520 corresponds to the third calculation step.

[0561] Next, the third processing unit 440 processes the aerodynamic sound data so that the volume of the aerodynamic sound attenuates as the distance D calculated by the third calculation unit 420 increases (S530). More specifically, the third processing unit 440 processes the aerodynamic sound data so that the volume of the aerodynamic sound attenuates at the γ power of the value obtained by dividing the representative wind speed by the wind speed at the ear. This step S530 corresponds to the third processing step.

[0562] The fourth output unit 450 outputs the aerodynamic sound data processed by the third processing unit 440 (S540). This step S540 corresponds to the fourth output step.

[0563] 33 is a flowchart of an eleventh operation of the acoustic signal processing device 400 according to this modification. Operation example 11 is an example in which the third processing unit 440 controls the frequency components of aerodynamic sound.

[0564] 33 , first, the fifth acquisition unit 410 acquires the fourth object audio information output by the information generating device 50, aerodynamic sound core information including data indicating the distribution of frequency components of aerodynamic sound, and second position information indicating the position of the listener L in the virtual space (S510f). This step S510f corresponds to the fifth acquisition step.

[0565] Next, the process of step S520 is performed.

[0566] Next, the third processing unit 440 processes the aerodynamic sound data so as to shift the distribution of the frequency components of the aerodynamic sound toward lower frequencies as the distance D calculated by the third calculation unit 420 increases (S530f). More specifically, the third processing unit 440 processes the aerodynamic sound data so as to shift the distribution of the frequency components of the aerodynamic sound toward frequencies that are the reciprocal multiple of the value (R1 above) obtained by dividing the representative wind speed by the wind speed at ear (S530f). This step S530f corresponds to the third processing step.

[0567] The fourth output unit 450 outputs the aerodynamic sound data processed by the third processing unit 440 (S540). This step S540 corresponds to the fourth output step.

[0568] The information generation method according to this modification can generate fourth object audio information that is associated with a generation position, a first wind direction, and a first estimated wind speed. The audio signal processing method according to this modification processes aerodynamic sound data so that, for example, the volume of the aerodynamic sound (second aerodynamic sound) decreases as the distance D between the object and the listener L increases. Furthermore, for example, the audio signal processing method according to this modification processes aerodynamic sound data so that the distribution of frequency components of the aerodynamic sound shifts toward lower frequencies as the distance D between the object and the listener L increases. Therefore, the aerodynamic sound (second aerodynamic sound) in the virtual space heard by the listener L resembles the aerodynamic sound (second aerodynamic sound) in the real space heard by the listener L. In other words, the listener L does not hear aerodynamic sound (second aerodynamic sound) that gives an unpleasant feeling, and the listener L can obtain a sense of realism.

[0569] (Seventh Modification of the Embodiment) The seventh modification of the embodiment will be described below. The following description will focus on the differences from the sixth modification of the embodiment, and the description of the commonalities will be omitted or simplified.

[0570] [Configuration] First, a description will be given of the configuration of an information generating device 60 and an acoustic signal processing device 500 according to Modification 7 of the present embodiment. Fig. 34 is a block diagram showing the functional configuration of the information generating device 60 and the acoustic signal processing device 500 according to this modification.

[0571] The information generating device 60 and the acoustic signal processing device 500 according to this modification, like Modification 6, can suppress the occurrence of the problem described in (The knowledge that forms the basis of this disclosure) that the listener L ends up hearing the second aerodynamic sound that causes discomfort. In this modification, like Modification 6, the object is an electric fan F.

[0572] In variant 6, a first wind W emitted from an electric fan F, which is an object in virtual space, is handled. In variant 7, a first wind and a second wind, which is a wind different from the first wind, are handled. As in variant 6, the first wind in this variant is the wind W emitted from an electric fan F, which is an object in virtual space. Furthermore, the second wind does not have to be a wind caused by an object in virtual space. In this variant, the second wind is a wind that occurs in nature in real space and is reproduced in virtual space (hereinafter referred to as natural wind). Furthermore, because the second wind is natural wind, the location of its generation cannot be identified in virtual space.

[0573] The information generating device 60 is a device that generates fourth object audio information related to the first wind and fifth object audio information related to the second wind, and outputs the information to the acoustic signal processing device 500. The acoustic signal processing device 500 is a device that acquires the output fourth object audio information and fifth object audio information, and outputs sound data to the headphones 20 based on the acquired fourth object audio information and fifth object audio information.

[0574] First, the information generating device 60 shown in FIG. 34 will be described.

[0575] The information generating device 60 includes a third receiving unit 61 , a seventh acquiring unit 62 , a fourth generating unit 63 , a sixth output unit 64 , a storage unit 65 , and a display unit 66 .

[0576] The third reception unit 61 receives an operation from a user (e.g., a creator of content executed in a virtual space) of the information generation device 60. Specifically, the third reception unit 61 is realized by a hardware button, but may also be realized by a touch panel or the like.

[0577] The seventh acquisition unit 62 acquires the generation position of a first wind blowing in the virtual space, the first wind direction and first estimated wind speed of the first wind, and the second wind direction and second estimated wind speed of a second wind blowing in the virtual space.

[0578] In this modification, the first wind blowing in the virtual space is wind W radiated from the object, electric fan F, as in modification 6. In other words, the position from which the first wind is generated is the position where the electric fan F is located. Note that in this modification, as in modification 6, it is sufficient that the first wind blows in the virtual space, and the object radiating this first wind, electric fan F, does not need to be located in the virtual space (more specifically, the virtual space in which listener L is located). In other words, as in modification 6, the object radiating the first wind, electric fan F, may or may not be located in the virtual space in which listener L is located.

[0579] The first wind direction is the wind direction of the first wind, which is the front direction of the wind W emitted from the object (electric fan F), for example, the direction Df shown in FIG. 19 . The first assumed wind speed according to this modification may be any value that indicates the wind speed of the first wind. Here, as in Modification 6, the first assumed wind speed is the wind speed of the first wind at a position a unit distance away from the generation position in the direction of the first wind direction, which is a reference distance. In other words, the first assumed wind speed is, for example, the wind speed wsF shown in FIG. 19 .

[0580] As described above, in the virtual space, a first wind, which is wind W emitted from the object, electric fan F, is blowing, and the listener L hears aerodynamic sound (second aerodynamic sound) generated when this wind W (first wind) reaches the listener L's ears.

[0581] Furthermore, the second wind is a natural wind, and the second wind direction is the direction of the second wind. For example, if the second wind is a south-southwest wind, the second wind direction indicates south-southwest. In this case, it goes without saying that the relationship between the geometric or mathematical directions indicating the directions in the virtual space and the geographical directions indicating north, south, east, and west must be determined in advance.

[0582] The second assumed wind speed is the speed of the second wind. Because the second wind is a natural wind, the second assumed wind speed is a constant value, independent of the location in the virtual space. In other words, the listener L is exposed to the second wind at a constant speed, independent of the position of the listener L in the virtual space.

[0583] Furthermore, since a second wind is blowing in the virtual space, the listener L hears aerodynamic sound (second aerodynamic sound) generated when the second wind reaches the listener L's ears.

[0584] Therefore, in this modified example, the listener L hears at least one of the second aerodynamic sound caused by the first wind and the second aerodynamic sound caused by the second wind.

[0585] In this modification, the third receiving unit 61 receives an operation from the user indicating the generation location, first wind direction, first estimated wind speed, second wind direction, and second estimated wind speed of the first wind. That is, the user operates the third receiving unit 61 to input the generation location, first wind direction, first estimated wind speed, second wind direction, and second estimated wind speed of the first wind, and the seventh obtaining unit 62 obtains the input generation location, first wind direction, first estimated wind speed, second wind direction, and second estimated wind speed. Details of the process by which the third receiving unit 61 receives an operation from the user indicating the first estimated wind speed and the second estimated wind speed will be described later using the processing of the display unit 66.

[0586] The third reception unit 61 may receive an operation from the user specifying that the unit distance is the second specified value. That is, the user operates the third reception unit 61 to set the unit distance to the second specified value.

[0587] The fourth generation unit 63 generates fourth object audio information in which the generation position, the first wind direction, and the first assumed wind speed acquired by the seventh acquisition unit 62 are associated with each other. The fourth generation unit 63 generates fifth object audio information in which the second wind direction and the second assumed wind speed acquired by the seventh acquisition unit 62 are associated with each other.

[0588] It is preferable that the third receiving unit 61 receives from the user an operation to specify directivity information indicating characteristics according to the direction of the first wind. The directivity information is the same as the information described in FIG. 19 and other figures. When such an operation is received, the fourth generating unit 63 generates fourth object audio information in which the generation position, the first wind direction, and the first estimated wind speed are associated with the directivity information indicated by the operation received by the third receiving unit 61.

[0589] The storage unit 65 is a storage device that stores computer programs and the like executed by the third reception unit 61, the seventh acquisition unit 62, the fourth generation unit 63, the sixth output unit 64, and the display unit 66. The storage unit 65 stores aerodynamic sound core information that includes a representative wind speed and aerodynamic sound data that indicates aerodynamic sound (second aerodynamic sound) that is generated when wind at the representative wind speed reaches the ears of the listener L in the virtual space.

[0590] In this modified example, the third reception unit 61 receives an operation indicating aerodynamic sound core information from the user. In other words, the aerodynamic sound core information is input by the user operating the third reception unit 61, and the input aerodynamic sound core information is stored in the storage unit 65.

[0591] The aerodynamic sound core information indicates a representative wind speed that shows an example of one wind speed value, and aerodynamic sound data that indicates the aerodynamic sound that is generated when wind at this representative wind speed reaches the ears of the listener L, and is a database used for information processing in the acoustic signal processing device 500, which will be described later. The aerodynamic sound data that indicates the aerodynamic sound (second aerodynamic sound) indicates the volume of the second aerodynamic sound, etc.

[0592] The sixth output unit 64 outputs the fourth object audio information generated by the fourth generation unit 63 and the fifth object audio information generated by the fourth generation unit 63. More specifically, when the generation position of the first wind is in the virtual space (more specifically, the virtual space where the listener L is located), the sixth output unit 64 outputs the fourth object audio information generated by the fourth generation unit 63. When the generation position of the first wind is not in the virtual space, the sixth output unit 64 outputs the fifth object audio information generated by the fourth generation unit 63. The sixth output unit 64 outputs the fourth object audio information or the fifth object audio information to the acoustic signal processing device 500. The sixth output unit 64 also outputs aerodynamic sound core information stored in the storage unit 65 to the acoustic signal processing device 500.

[0593] The display unit 66 is a display device that displays an image in which wind speeds are associated with words that express the wind speeds. The display unit 30 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro Luminescence) panel.

[0594] FIG. 35 is a diagram showing an example of an image displayed on the display unit 66 according to this modification. This image is an image in which wind speed and a word expressing that wind speed are linked. For example, when the wind speed is 0.0-0.2 m / s, the image shows that the word expressing that wind speed is "calm." Also, an explanation of the wind "on land" and "on sea" when wind of that speed is blowing is displayed. Furthermore, the image may include text, figures, illustrations, etc.

[0595] The user visually recognizes the image displayed on display unit 66. Then, third reception unit 61 receives from the user a first operation for specifying the wind speed indicated by the displayed image as the first estimated wind speed, and a second operation for specifying the wind speed indicated by the displayed image as the second estimated wind speed. In other words, the user operates third reception unit 61 to input the first estimated wind speed and the second estimated wind speed, and seventh acquisition unit 62 acquires the input first estimated wind speed and second estimated wind speed.

[0596] Next, an acoustic signal processing device 500 shown in FIG. 34 will be described.

[0597] When the information generating device 60 outputs fourth object audio information, the audio signal processing device 500 processes the second aerodynamic sound caused by the first wind related to the fourth object audio information. When the information generating device 60 outputs fifth object audio information, the audio signal processing device 500 processes the second aerodynamic sound caused by the second wind related to the fifth object audio information.

[0598] As shown in FIG. 34 , the acoustic signal processing device 500 includes an eighth acquisition unit 510 , a third calculation unit 420 , a fourth processing unit 540 , a seventh output unit 550 , and a storage unit 570 .

[0599] The eighth acquirer 510 acquires the fourth object audio information or the fifth object audio information output by the information generator 60. The eighth acquirer 510 acquires second position information indicating the position of the listener L in the virtual space and the aerodynamic sound core information output by the information generator 60. The eighth acquirer 510 acquires the second position information from the headphones 20 (more specifically, the head sensor unit 21), but is not limited to this.

[0600] When the eighth acquirer 510 acquires the fourth object audio information, that is, when the generation position of the first wind is in the virtual space (more specifically, the virtual space where the listener L is located), the third calculator 420 performs the following process. That is, the third calculator 420 calculates the distance D between the generation position (that is, the electric fan F) and the listener L based on the generation position included in the fourth object audio information acquired by the eighth acquirer 510 and the acquired second position information. The third calculator 420 calculates the distance D in the same manner as the first calculator 120 according to the embodiment.

[0601] Furthermore, the third calculation unit 420 calculates the point-to-point direction connecting the object (fan F) and the listener L based on the generation position included in the fourth object audio information acquired by the eighth acquisition unit 510 and the acquired second position information.

[0602] When the eighth acquisition unit 510 acquires fourth object audio information, the fourth processing unit 540 processes the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the second position information acquired by the eighth acquisition unit 510. In other words, in this case, processing is performed on the second aerodynamic sound caused by the first wind. More specifically, like the third processing unit 440 according to Modification 6, the fourth processing unit 540 processes the aerodynamic sound data so that the volume of the aerodynamic sound attenuates as the distance D calculated by the third calculation unit 420 increases. The distance D calculated by the third calculation unit 420 is a value that depends on the position indicated by the second position information. In this case, the aerodynamic sound data processed by the fourth processing unit 540 is data indicating the second aerodynamic sound caused by the first wind.

[0603] When the eighth acquisition unit 510 acquires the fifth object audio information, the fourth processing unit 540 processes the aerodynamic sound data included in the aerodynamic sound core information without being based on the position indicated by the second position information acquired by the eighth acquisition unit 510. Because the second wind is a natural wind, the second estimated wind speed of the second wind indicates a constant value, independent of the position of the listener L indicated by the second position information. Therefore, the fourth processing unit 540 processes the aerodynamic sound data without being dependent on the position indicated by the second position information.

[0604] For example, when the fifth object audio information is acquired, the fourth processing unit 540 performs the following process.

[0605] If the wind speed at which the second wind reaches the ear of the listener L is Se2, Se2 satisfies (Equation 14).

[0606] Se2 = second expected wind speed (Equation 14)

[0607] Furthermore, the value obtained by dividing the representative wind speed by the ear arrival wind speed Se2 is defined as R2. Furthermore, V5, which is the volume of the second aerodynamic sound heard by the listener L, satisfies (Equation 15).

[0608] V5 = volume indicated by the aerodynamic noise data of the aerodynamic noise core information × {(1 / R2)}^γ (Equation 15)

[0609] The storage unit 570 is a storage device that stores computer programs executed by the eighth acquisition unit 510, the third calculation unit 420, the fourth processing unit 540, and the seventh output unit 550, etc.

[0610] That is, in this modification, the first wind, which is the wind W emitted from the object, is processed according to the position of the listener L. However, the second wind, which is natural wind, is processed not according to the position of the listener L but independent of the position of the listener L.

[0611] Below, a twelfth operational example of the information generating method performed by the information generating device 60 and a thirteenth operational example of the acoustic signal processing method performed by the acoustic signal processing device 500 will be described.

[0612] [Operation Example 12] FIG. 36 is a flowchart of operation example 12 of the information generating device 60 according to this modification.

[0613] As shown in FIG. 36, first, the display unit 66 displays an image in which the wind speed is associated with a word expressing the wind speed (S610).

[0614] Next, the third reception unit 61 receives from the user a first operation for specifying the wind speed indicated by the displayed image as the first estimated wind speed, and a second operation for specifying the wind speed indicated by the displayed image as the second estimated wind speed (S620). At this time, the third reception unit 61 may receive from the user operations for specifying the generation position of the first wind, the first wind direction, and the second wind direction. In other words, the user operates the third reception unit 61 to input the generation position of the first wind, the first wind direction, the first estimated wind speed, the second wind direction, and the second estimated wind speed. This step S620 corresponds to a third reception step.

[0615] Next, the seventh acquisition unit 62 acquires the generation position of the first wind, the first wind direction of the first wind, the first estimated wind speed which is the wind speed of the first wind, the second wind direction of the second wind, and the second estimated wind speed which is the wind speed of the second wind (S630). Here, the seventh acquisition unit 62 acquires the generation position of the first wind, the first wind direction, the first estimated wind speed, the second wind direction, and the second estimated wind speed which were input in step S620. This step S630 corresponds to the seventh acquisition step.

[0616] Next, the fourth generation unit 63 generates fourth object audio information in which the generation position, the first wind direction, and the first assumed wind speed are associated with each other, and also generates fifth object audio information in which the second wind direction and the second assumed wind speed are associated with each other (S640). This step S640 corresponds to a fourth generation step.

[0617] Next, the storage unit 65 stores aerodynamic sound core information including the representative wind speed and aerodynamic sound data indicating the aerodynamic sound (second aerodynamic sound) that is generated when wind at the representative wind speed reaches the ears of the listener L (S650). This step S650 corresponds to the storage step.

[0618] The sixth output unit 64 outputs the fourth object audio information generated by the fourth generation unit 63 if the generation position of the first wind is within the virtual space, and outputs the fifth object audio information generated by the fourth generation unit 63 if the generation position of the first wind is not within the virtual space (S660). At this time, the sixth output unit 64 may also output the aerodynamic sound core information stored in the storage unit 65. Step S660 corresponds to a sixth output step.

[0619] [Operation Example 13] FIG. 37 is a flowchart of operation example 13 of the acoustic signal processing device 500 according to this modification.

[0620] 37 , first, the eighth acquisition unit 510 acquires the second position information indicating the position of the listener L in the virtual space, and the fourth object audio information or the fifth object audio information output by the information generating device 60 (S710). At this time, the eighth acquisition unit 510 may also acquire aerodynamic sound core information. This step S710 corresponds to the eighth acquisition step.

[0621] Next, when the eighth acquirer 510 acquires the fourth object audio information, the third calculator 420 performs the following process. That is, the third calculator 420 calculates the distance D between the generation position included in the fourth object audio information acquired by the eighth acquirer 510 and the acquired second position information, based on the generation position included in the fourth object audio information acquired by the eighth acquirer 510 and the acquired second position information (S720). At this time, the third calculator 420 calculates the direction between the two points connecting the object (electric fan F) and the listener L, based on the generation position included in the fourth object audio information acquired by the eighth acquirer 510 and the acquired second position information.

[0622] Next, when fourth object audio information is acquired, the fourth processing unit 540 processes the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the acquired second position information, and when fifth object audio information is acquired, it processes the aerodynamic sound data included in the aerodynamic sound core information without being based on the position indicated by the acquired second position information (S730). This step S730 corresponds to the fourth processing step.

[0623] The seventh output unit 550 outputs the aerodynamic sound data processed by the fourth processing unit 540 (S740). This step S740 corresponds to the seventh output step.

[0624] The information generation method according to this modification can generate fourth object audio information in which a generation position, a first wind direction, and a first assumed wind speed are associated, and fifth object audio information in which a second wind direction and a second assumed wind speed are associated. The audio signal processing method according to this modification processes the aerodynamic sound data based on the position indicated by the second position information, so that the aerodynamic sound caused by the first wind in the virtual space heard by the listener L (second aerodynamic sound) resembles the aerodynamic sound caused by the first wind in the real space heard by the listener L (second aerodynamic sound). Furthermore, the audio signal processing method according to this modification processes the aerodynamic sound data not based on the position indicated by the second position information, so that the aerodynamic sound caused by the second wind in the virtual space heard by the listener L (second aerodynamic sound) resembles the aerodynamic sound caused by the second wind in the real space heard by the listener L (second aerodynamic sound). That is, the listener L does not hear the aerodynamic sound (second aerodynamic sound) that gives the listener an uncomfortable feeling, and the listener L can obtain a sense of realism.

[0625] (Eighth Modification of the Embodiment) The eighth modification of the embodiment will be described below. The following description will focus on the differences from the seventh modification of the embodiment, and description of commonalities will be omitted or simplified.

[0626] [Configuration] First, a description will be given of the configuration of an information generating device 70 and an acoustic signal processing device 500 according to Modification 8 of the present embodiment. Fig. 38 is a block diagram showing the functional configuration of the information generating device 70 and the acoustic signal processing device 500 according to this modification.

[0627] In this modification, an acoustic signal processing device 500 according to modification 7 is used. The information generating device 70 according to this modification has the same configuration as the information generating device 70 according to modification 7, except that the information generating device 70 according to this modification includes a sixth acquiring unit 72, a third generating unit 73, and a fifth output unit 74 instead of the seventh acquiring unit 62, the fourth generating unit 63, and the sixth output unit 64, and does not include the display unit 66.

[0628] In the seventh modification, the first wind and the second wind, which is a natural wind, are handled. In this modification, the first wind is not handled, but the second wind, which is a natural wind, is handled.

[0629] The information generating device 70 is a device that generates fifth object audio information related to the second wind and outputs it to the acoustic signal processing device 500. The acoustic signal processing device 500 is a device that acquires the output fifth object audio information and outputs sound data to the headphones 20 based on the acquired fifth object audio information.

[0630] First, the information generating device 70 shown in FIG. 38 will be described.

[0631] The information generating device 70 includes a third receiving unit 61 , a sixth obtaining unit 72 , a third generating unit 73 , a fifth output unit 74 , and a storage unit 75 .

[0632] The third reception unit 61 receives an operation from a user (e.g., a creator of content executed in a virtual space) of the information generation device 70. Specifically, the third reception unit 61 is realized by a hardware button, but may also be realized by a touch panel or the like.

[0633] The sixth acquisition unit 72 acquires a second wind direction and a second estimated wind speed of a second wind blowing in the virtual space.

[0634] In this modified example, since a second wind is blowing in the virtual space, the listener L hears aerodynamic sound (second aerodynamic sound) generated when the second wind reaches the listener L's ears.

[0635] The third receiving unit 61 receives an operation indicating the second wind direction and the second estimated wind speed from the user. That is, the second wind direction and the second estimated wind speed are input by the user operating the third receiving unit 61, and the sixth obtaining unit 72 obtains the input second wind direction and second estimated wind speed.

[0636] The third generating unit 73 generates fifth object audio information in which the second wind direction and the second estimated wind speed acquired by the sixth acquiring unit 72 are associated with each other.

[0637] The storage unit 75 is a storage device that stores computer programs and the like executed by the third reception unit 61, the sixth acquisition unit 72, the third generation unit 73, and the fifth output unit 74. The storage unit 75 stores aerodynamic sound core information that includes a representative wind speed and aerodynamic sound data that indicates aerodynamic sound (second aerodynamic sound) that is generated when wind at the representative wind speed reaches the ears of the listener L in the virtual space.

[0638] The fifth output unit 74 outputs the fifth object audio information generated by the third generation unit 73 and the aerodynamic sound core information stored in the storage unit 75 to the acoustic signal processing device 500 .

[0639] Next, an acoustic signal processing device 500 shown in FIG. 38 will be described.

[0640] The acoustic signal processing device 500 according to this modification performs processing on the second aerodynamic sound caused by the second wind associated with the fifth object audio information.

[0641] The eighth acquisition unit 510 acquires the fifth object audio information and the aerodynamic sound core information output by the information generation device 70 .

[0642] The fourth processing unit 540 processes the aerodynamic sound data included in the aerodynamic sound core information without being based on the position indicated by the second position information acquired by the eighth acquisition unit 510. The fourth processing unit 540 performs the same processing as when the fifth object audio information is acquired in Modification 7.

[0643] An operation example 14 of the information generating method performed by the information generating device 70 will be described below.

[0644] [Operation Example 14] FIG. 39 is a flowchart of operation example 14 of the information generating device 70 according to this modification.

[0645] As shown in FIG. 39, first, the third receiving unit 61 receives an operation indicating a second wind direction and a second estimated wind speed from the user (S810).

[0646] Next, the sixth acquisition unit 72 acquires the second wind direction of the second wind and the second estimated wind speed, which is the wind speed of the second wind (S820). Here, the sixth acquisition unit 72 acquires the second wind direction and second estimated wind speed input in step S810. This step S820 corresponds to the sixth acquisition step.

[0647] Next, the third generating unit 73 generates fifth object audio information in which the second wind direction and the second estimated wind speed are associated with each other (S830). This step S830 corresponds to a third generating step.

[0648] Next, the storage unit 75 stores aerodynamic sound core information including the representative wind speed and aerodynamic sound data indicating the aerodynamic sound generated when wind at the representative wind speed reaches the ears of the listener L (S840). This step S840 corresponds to the storage step.

[0649] The fifth output unit 74 outputs the fifth object audio information generated by the third generation unit 73 (S850). At this time, the fifth output unit 74 may also output the aerodynamic sound core information stored in the storage unit 75. This step S850 corresponds to a fifth output step.

[0650] Further processing is performed by the acoustic signal processing device 500, and the listener L hears the second aerodynamic sound caused by the second wind.

[0651] In all of the above-described embodiments, the assumed wind speed was always treated as a given constant value. However, in real space, wind speed, particularly the wind speed of natural wind, fluctuates slowly. Therefore, if aerodynamic sounds are generated in a virtual space assuming a constant assumed wind speed, it would be somewhat unnatural from a practical perspective. Therefore, for example, if the assumed wind speed is given as S, realism can be enhanced by varying the assumed wind speed slowly and irregularly around S. Even in this case, the process of generating aerodynamic sounds can be performed according to the method described in the above embodiments, assuming that the ever-changing wind speed is the assumed wind speed at that moment. The same applies to wind direction. In particular, a constant wind direction can cause unnaturalness in natural winds. Therefore, a volume ratio, phase difference, time difference, or the like may be applied between the aerodynamic sound signals output for the left ear and the aerodynamic sound signals output for the right ear, thereby expressing a fluctuating wind direction.

[0652] Next, the pipeline processing will be described.

[0653] Some or all of the processing performed by the above-described acoustic signal processing device 100 may be perfo...

Claims

1. a first acquisition step of acquiring object information including first position information indicating a position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a processing means for the first sound data, and second position information indicating a position of a listener of the first sound in the virtual space; a first calculation step of calculating a distance between the object and the listener based on the first position information included in the acquired object information and the second position information acquired; a determining step of determining a processing means for processing the first sound data from among a first processing means for processing a volume according to the calculated distance based on the first identification information included in the acquired object information, and a second processing means for processing a volume according to the calculated distance in a manner different from that of the first processing means; a first processing step of processing the first sound data by the determined processing means; and a first output step of outputting the processed first sound data. Acoustic signal processing method.

2. the first processing means processes the first sound data such that a volume of the first sound data is attenuated inversely proportional to an increase in the calculated distance; The second processing means processes the first sound data so that the volume increases or decreases in a manner different from that of the first processing means as the calculated distance increases.

2. The method of claim 1.

3. The second processing means processes the first sound data so that the volume attenuates as the xth power (x≠1) of the distance.

3. The method of claim 2.

4. The first identification information indicates that the processing means for the first sound data is the second processing means, and indicates the value of x. The method of processing an acoustic signal according to claim 3.

5. A second acquisition step of acquiring first sound data indicating a first sound and first position information indicating a position of an object in a virtual space; a first generation step of generating, from the acquired first sound data and the acquired first position information, first object audio information including information about the object that reproduces the first sound generated at the position of the listener due to the object and the first position information. Information generation method.

6. The object emits wind; The listener is exposed to the emitted wind, The first sound is an aerodynamic sound generated when the wind emitted from the object reaches the ear of the listener. The information generating method according to claim 5 .

7. In the first generating step, the first object audio information is generated, the first object audio information further including unit distance information; The unit distance information is information including a unit distance that is a reference distance, and aerodynamic sound data that indicates the aerodynamic sound at a position that is away from the position of the object by the unit distance. The information generating method according to claim 6.

8. In the first generating step, the first object audio information further includes directivity information; The directivity information is information indicating characteristics according to a direction of the radiated wind, The aerodynamic sound data indicated in the unit distance information is data indicating the aerodynamic sound at a position away from the position of the object indicated in the directivity information by the unit distance in a front direction in which the object radiates the wind. The information generating method according to claim 7.

9. In the first generation step, the first object audio information further includes flag information indicating whether or not to perform a process of convolving a head related transfer function that depends on a direction from which a sound arrives on a first sound signal based on the first sound data indicating the first sound generated from the object during playback of the first sound. The information generating method according to claim 5 .

10. a fourth acquisition step of acquiring a generation position of a first wind blowing in a virtual space, a first wind direction of the first wind, and a first assumed wind speed which is a wind speed of the first wind; a second generation step of generating fourth object audio information in which the acquired generation position, the first wind direction, and the first assumed wind speed are associated with each other; a storage step of storing aerodynamic sound core information including a representative wind speed and aerodynamic sound data indicating aerodynamic sound generated when wind at the representative wind speed reaches the ears of a listener in the virtual space; and a third output step of outputting the generated fourth object audio information and the stored aerodynamic sound core information. Information generation method.

11. The first assumed wind speed is a wind speed of the first wind at a position away from the generation position in the first wind direction by a unit distance that is a reference distance. The information generating method according to claim 10.

12. A second reception step of receiving, from a user, an operation for designating directivity information indicating a characteristic according to a direction of the first wind, In the second generating step, the fourth object audio information is generated in which the acquired generation position, the first wind direction, and the first assumed wind speed are associated with the directivity information indicated by the received operation. The information generating method according to claim 11.

13. a fifth acquisition step of acquiring the fourth object audio information and the aerodynamic sound core information output by the information generation method according to claim 11, and second position information indicating a position of the listener in the virtual space; a third calculation step of calculating a distance between the generation position and the listener based on the generation position included in the acquired fourth object audio information and the acquired second position information; a third processing step of processing the aerodynamic sound data so that the volume of the aerodynamic sound decreases as the calculated distance increases; and a fourth output step of outputting the processed aerodynamic sound data. Acoustic signal processing method.

14. In the third processing step, the aerodynamic sound data is processed based on an ear arrival wind speed, which is the wind speed of the first wind when it reaches the ear of the listener, The wind speed reaching the ear decreases as the calculated distance increases. The method of processing an acoustic signal according to claim 13.

15. The ear reaching wind speed is a value attenuated by the zth power of the value obtained by dividing the calculated distance by the unit distance.

15. The method of processing an acoustic signal according to claim 14.

16. a fifth acquisition step of acquiring the fourth object audio information output by the information generation method according to claim 11, the aerodynamic sound core information including data indicating a distribution of frequency components of the aerodynamic sound, and second position information indicating a position of the listener in the virtual space; a third calculation step of calculating a distance between the generation position and the listener based on the generation position included in the acquired fourth object audio information and the acquired second position information; a third processing step of processing the aerodynamic sound data so as to shift the distribution of the frequency components of the aerodynamic sound toward a lower frequency side as the calculated distance increases; and a fourth output step of outputting the processed aerodynamic sound data. Acoustic signal processing method.

17. In the third processing step, the aerodynamic sound data is processed based on an ear arrival wind speed, which is the wind speed of the first wind when it reaches the ear of the listener, The wind speed reaching the ear decreases as the calculated distance increases.

17. The method of processing an acoustic signal according to claim 16.

18. The ear reaching wind speed is a value attenuated by the zth power of the value obtained by dividing the calculated distance by the unit distance.

18. A method for processing an acoustic signal according to claim 17.

19. a first acquisition unit that acquires object information including first position information indicating a position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a processing means for the first sound data, and second position information indicating a position of a listener of the first sound in the virtual space; a first calculation unit that calculates a distance between the object and the listener based on the first position information included in the acquired object information and the second position information; a determination unit that determines which of a first processing means processes the first sound data from among a first processing means that processes a volume according to the calculated distance based on the first identification information included in the acquired object information and a second processing means that processes a volume according to the calculated distance in a manner different from that of the first processing means; a first processing unit that processes the first sound data by the determined processing means; a first output unit that outputs the processed first sound data. Acoustic signal processing device.

20. A computer program for causing a computer to execute the acoustic signal processing method according to any one of claims 1 to 4 and 13 to 18.

21. A computer program for causing a computer to execute the information generation method described in any one of claims 5 to 12.