Acoustic signal processing method, computer program, and acoustic signal processing device

Abstract

An acoustic signal processing method includes: an acquisition step of acquiring first position information indicating a position of an object that is a moving body in a virtual space, and second position information indicating a position of a listener in the virtual space; a first calculation step for calculating the moving speed of the object on the basis of the acquired first position information; a second calculation step for calculating the distance between the object and the listener on the basis of the acquired first position information and the acquired second position information; a generation step for generating, on the basis of the calculated movement speed and the calculated distance, an aero-sound signal indicating an aero-sound generated by the wind generated by the movement of the subject reaching the ear of the listener; and an output step of outputting the generated pneumatic sound signal.

Description

Technical Field

[0001] The present disclosure relates to an audio signal processing method and the like. Background Art

[0002] Patent document 1 discloses a technique related to a three-dimensional sound field spatial reproduction device as an acoustic signal processing device having a Doppler calculation mechanism for performing Doppler effect processing on a convoluted acoustic signal based on the movement speed of a listener and the movement speed of a sound source.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 07-312800

[0006] Patent Document 2: International Publication No. 2021 / 180938 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, in the technology disclosed in Patent Document 1, it is sometimes difficult to provide a sense of presence to the listener.

[0009] Therefore, an object of the present disclosure is to provide an audio signal processing method and the like that can provide a sense of presence to a listener.

[0010] Means used to solve problems

[0011] A sound signal processing method according to a technical solution of the present disclosure includes: an acquisition step of acquiring first position information indicating the position of an object as a moving body in a virtual space, and second position information indicating the position of a listener in the virtual space; a first calculation step of calculating the moving speed of the object based on the first position information obtained; a second calculation step of calculating the distance between the object and the listener based on the first position information obtained and the second position information obtained; a generation step of generating an aerodynamic sound signal based on the calculated moving speed and the calculated distance, the aerodynamic sound signal indicating the aerodynamic sound generated by the wind generated by the movement of the object reaching the ears of the listener; and an output step of outputting the generated aerodynamic sound signal.

[0012] In addition, an audio signal processing method related to a technical solution of the present disclosure includes: a generation step, processing a noise signal separately using multiple frequency band emphasis filters, synthesizing the multiple processed noise signals, thereby generating an aerodynamic sound signal, which represents the aerodynamic sound generated by the wind reaching the ears of the listener in the virtual space; and an output step, outputting the generated aerodynamic sound signal, wherein the multiple frequency band emphasis filters are filters used to simulate the aerodynamic sound generated by the shape of the ears or head of the listener.

[0013] Furthermore, a program according to one technical aspect of the present disclosure causes a computer to execute the above-mentioned sound signal processing method.

[0014] In addition, an audio signal processing device related to a technical solution of the present disclosure includes: an acquisition unit, which acquires first position information indicating the position of an object as a moving body in a virtual space, and second position information indicating the position of a listener in the above virtual space; a first calculation unit, which calculates the moving speed of the above object based on the acquired first position information; a second calculation unit, which calculates the distance between the above object and the above listener based on the acquired first position information and the acquired second position information; a generation unit, which generates an aerodynamic sound signal based on the calculated moving speed and the calculated distance, the aerodynamic sound signal indicating the aerodynamic sound generated by the wind generated by the movement of the above object reaching the ears of the above listener; and an output unit, which outputs the generated aerodynamic sound signal.

[0015] In addition, these inclusive or specific technical solutions can also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0016] Effects of the Invention

[0017] According to an audio signal processing method according to a technical solution of the present disclosure, a sense of presence can be brought to the listener. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an example of a diagram for explaining aerodynamic sound generated by reaching the head or ears of a listener.

[0019] Figure 2A This is another example of a diagram for explaining the aerodynamic sound generated by reaching the head or ears of the listener.

[0020] Figure 2BThis is a diagram showing an immersive audio reproduction system as an example of a system to which the audio processing or decoding processing of the present disclosure can be applied.

[0021] Figure 2C This is a functional block diagram showing the configuration of an encoding device as an example of the encoding device of the present disclosure.

[0022] Figure 2D This is a functional block diagram showing the configuration of a decoding device as an example of the decoding device of the present disclosure.

[0023] Figure 2E This is a functional block diagram showing the configuration of an encoding device as another example of the encoding device of the present disclosure.

[0024] Figure 2F This is a functional block diagram showing the configuration of a decoding device which is another example of the decoding device of the present disclosure.

[0025] Figure 2G It means as Figure 2D or Figure 2F A functional block diagram of the structure of a decoder of an example of a decoder in FIG.

[0026] Figure 2H It means as Figure 2D or Figure 2F A functional block diagram of the structure of another example of a decoder in the decoder.

[0027] Fig.2I This is a diagram showing an example of the physical structure of the sound signal processing device.

[0028] Figure 2J This is a diagram showing an example of the physical structure of an encoding device.

[0029] Figure 3A This is a block diagram showing the functional structure of the sound signal processing device according to the embodiment.

[0030] Figure 3B This is a flowchart of an example of the operation of the selection unit according to the embodiment.

[0031] Figure 4 This is a flowchart of operation example 1 of the sound signal processing device according to the embodiment.

[0032] Figure 5 It means in Figure 4 The flowchart of the first example of a method in which the generator generates a pneumatic sound signal in step S50 is shown.

[0033] Figure 6 It means in Figure 4The flowchart of the first example of a method in which the generator generates a pneumatic sound signal in step S50 is shown.

[0034] Figure 7 1 is a schematic diagram of a head-mimicking microphone for collecting aerodynamic sound data according to the method of the first example of operation example 1.

[0035] Figure 8 FIG. 1 is a schematic diagram of another human head-like microphone for collecting aerodynamic sound data according to the method of Example 1 of Operation Example 1. FIG.

[0036] Fig. 9 It means in Figure 4 The flowchart of the method of the second example in which the generator generates the pneumatic sound signal in step S50 shown in FIG.

[0037] Fig. 10A This is a diagram for explaining a process of generating a pneumatic sound signal by a generation unit of the method of the second example of Operation Example 1.

[0038] Fig. 10B This is another diagram for explaining the process of generating a pneumatic sound signal by the generation unit of the method of the second example of Operation Example 1.

[0039] Fig.11 This is a diagram showing an example of four bandpass filters and their processing in the method of the second example of Operation Example 1.

[0040] Fig.12 This is another diagram for explaining the process of generating a processing signal by the generation unit of the method of the second example of the first operation example.

[0041] Fig.13 This is another diagram for explaining the process of generating a processing signal by the generation unit of the method of the second example of the first operation example.

[0042] Fig.14 This is a flowchart of operation example 2 of the sound signal processing device according to the embodiment.

[0043] Fig.15 This is a flowchart of operation example 3 of the sound signal processing device according to the embodiment.

[0044] Fig.16 This is a flowchart of operation example 4 of the sound signal processing device according to the embodiment.

[0045] Fig.17 This is a diagram showing an aircraft serving as a target of Operation Example 4 of the acoustic signal processing device according to the embodiment.

[0046] Fig.18 This is a diagram showing a round fan as a target of operation example 5 of the sound signal processing device according to the embodiment.

[0047] Fig.19 This is a flowchart of operation example 5 of the sound signal processing device according to the embodiment.

[0048] Fig. 20 It is used to illustrate Figure 2G and Figure 2H A diagram showing an example of a functional block diagram and steps in which a rendering unit performs pipeline processing. DETAILED DESCRIPTION

[0049] (Understanding that is the basis of the present disclosure)

[0050] Conventionally, there is known an acoustic signal processing device that processes an acoustic signal (ie, a sound signal) in order to exert the Doppler effect in a virtual space.

[0051] Patent document 1 discloses a technique related to a three-dimensional sound field spatial reproduction device as an acoustic signal processing device. The acoustic signal processing device includes a Doppler operation mechanism that performs Doppler effect processing on an acoustic signal after convolution operation based on the moving speed of a listener and the moving speed of a sound source (i.e., an object that emits sound). Thus, by the listener listening to the sound processed by the Doppler effect, the listener can recognize that the object that emits the sound (e.g., an ambulance emitting a siren) is moving.

[0052] In addition, although not disclosed in Patent Document 1, even if an object does not make a sound, when the object moves and generates wind noise, the listener can hear the wind noise. Thus, the listener can more strongly recognize that the object is moving. For example, in the case where the object is a baseball bat, when the object (bat) moves (changes position), that is, when the bat is swung, wind noise is generated. By listening to the wind noise, the listener can recognize that the bat is swung.

[0053] Such Doppler effect-processed sound and wind noise are used in applications such as virtual reality (VR (Virtual Reality)) and augmented reality (AR (Augmented Reality)) to reproduce three-dimensional sound in a space where a user exists.

[0054] Here, an object that does not make any sound and does not move at such a high speed as to generate wind noise is studied.

[0055] Even such an object may generate wind by moving in the air, and the wind may reach the listener. In this case, the wind reaches the listener's head or ears, for example, and generates aerodynamic sound. Aerodynamic sound is also the sound generated by the wind reaching the listener's ears in the virtual space, and is also the sound generated by the shape of the listener's ears or head. The listener perceives that the object is moving by hearing the aerodynamic sound. Figure 1 and Figure 2A This example will be described.

[0056] Figure 1 This is an example of a diagram for explaining aerodynamic sound generated by reaching the head or ears of the listener L. Figure 1 The illustrated vehicle 400 does not generate any sound such as a whistle, and does not move at such a high speed as to generate wind shear sound.

[0057] Even such a vehicle 400 generates wind W by the movement of the vehicle 400. When the wind W reaches the listener L, for example, aerodynamic sound corresponding to the shape of the ear of the listener L is generated, and the listener L hears the aerodynamic sound. Therefore, the listener L can perceive the movement of the vehicle 400 through the sense of hearing.

[0058] Figure 2A This is another example of a diagram for explaining the aerodynamic sound generated by reaching the head or ears of the listener L. Figure 2A The electric fan 600 shown has a plurality of large blades 601 that rotate.

[0059] When the blade 601 rotates and does not rotate at a high speed to generate wind noise, wind W is generated. When the wind W reaches the listener L, for example, aerodynamic sound corresponding to the shape of the ear of the listener L is generated, and the listener L hears the aerodynamic sound. Therefore, the listener L can perceive that the blade 601 is rotating (moving) through hearing.

[0060] If the listener L can perceive that the object (the vehicle 400 or the blade 601 ) is moving due to such aerodynamic sound, the listener L can obtain a better sense of presence.

[0061] The technology disclosed in Patent Document 1 does not indicate that the aerodynamic sound generated by the wind W reaching the head or ears of the listener L is output to the listener L. That is, in the conventional technology (the technology shown in Patent Document 1) that cannot output the aerodynamic sound, it is difficult to provide the listener L with a sense of presence.

[0062] Therefore, an acoustic signal processing method or the like that can provide the listener L with a sense of presence is required.

[0063] Therefore, the sound signal processing method of the first technical solution of the present disclosure includes: an acquisition step, obtaining first position information indicating the position of an object as a moving body in a virtual space, and second position information indicating the position of a listener in the above virtual space; a first calculation step, calculating the moving speed of the above object based on the above first position information obtained; a second calculation step, calculating the distance between the above object and the above listener based on the above first position information obtained and the above second position information obtained; a generation step, generating an aerodynamic sound signal based on the calculated moving speed and the calculated distance, the aerodynamic sound signal indicating the aerodynamic sound generated by the wind generated by the movement of the above object reaching the ears of the above listener; and an output step, outputting the generated aerodynamic sound signal.

[0064] Thus, the sound signal processing method can generate an aerodynamic sound signal representing the aerodynamic sound generated by the wind generated by the movement of the object reaching the ears of the listener based on the position of the object in the virtual space and the position of the listener, and output the generated aerodynamic sound signal. The aerodynamic sound signal is output to headphones or speakers, etc., and the listener can hear the aerodynamic sound. Therefore, the listener can perceive that the object is moving and can get a sense of presence. That is, the sound signal processing method related to a technical solution of the present disclosure can bring a sense of presence to the listener.

[0065] In addition, for example, the sound signal processing method related to the second technical solution of the present disclosure is the sound signal processing method related to the first technical solution. In the above-mentioned generation step, the above-mentioned pneumatic sound signal is generated in a manner such that the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, and the volume of the above-mentioned pneumatic sound is increased as the calculated distance is smaller.

[0066] Thus, the aerodynamic sound heard by the listener can be controlled according to the moving speed of the object and the distance between the object and the listener, so the listener can get a better sense of presence. In other words, the acoustic signal processing method can give the listener a better sense of presence.

[0067] In addition, for example, the sound signal processing method related to the third technical solution of the present disclosure is the sound signal processing method related to the first or second technical solution. In the above-mentioned acquisition step, the ability information indicating whether the above-mentioned object can generate the above-mentioned pneumatic sound is acquired; in the above-mentioned generation step, the above-mentioned pneumatic sound signal is generated when the above-mentioned ability information acquired indicates that the above-mentioned object generates the above-mentioned pneumatic sound.

[0068] Thus, the pneumatic sound signal is generated only when the information indicates that the object can generate pneumatic sound. That is, the acoustic signal processing method does not need to generate a pneumatic sound signal when the information does not indicate that the object can generate pneumatic sound, so the processing load for generating pneumatic sound can be suppressed in the acoustic signal processing method.

[0069] In addition, for example, the sound signal processing method related to the fourth technical solution of the present disclosure is the sound signal processing method related to the first or second technical solution, and in the above-mentioned acquisition step, shape information representing the shape of the above-mentioned object is acquired; in the above-mentioned generation step, the above-mentioned aerodynamic sound signal is generated based on the above-mentioned shape represented by the acquired shape information, the calculated above-mentioned moving speed and the calculated above-mentioned distance.

[0070] Thus, an aerodynamic sound signal is generated according to the shape of the object, the moving speed of the object, and the distance between the object and the listener, so the listener can hear the aerodynamic sound with a better sense of presence. In other words, the acoustic signal processing method can give the listener a better sense of presence.

[0071] In addition, for example, the sound signal processing method related to the fifth technical solution of the present disclosure is the sound signal processing method related to the fourth technical solution, and in the above-mentioned generation step, the above-mentioned pneumatic sound signal is generated in a manner such that the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, and the volume of the above-mentioned pneumatic sound is increased as the calculated distance is smaller, and based on the above-mentioned shape information obtained, the volume of the above-mentioned pneumatic sound is increased as the volume of the above-mentioned object is greater.

[0072] Thus, the aerodynamic sound heard by the listener can be controlled according to the moving speed of the object, the distance between the object and the listener, and the volume of the object, so the listener can get a better sense of presence. That is, the sound signal processing method can bring a better sense of presence to the listener.

[0073] In addition, for example, the sound signal processing method related to the 6th technical solution of the present disclosure is the sound signal processing method related to the 4th technical solution, including a third calculation step, in which the wind volume per unit time generated when the above-mentioned object moves is calculated based on the calculated moving speed and the above-mentioned shape information obtained; in the above-mentioned generation step, the above-mentioned aerodynamic sound signal is generated based on the calculated moving speed, the calculated distance and the calculated wind volume.

[0074] Thus, the aerodynamic sound signal is generated according to the moving speed of the object, the distance between the object and the listener, and the wind volume, so the listener can hear the aerodynamic sound with a better sense of presence. That is, the acoustic signal processing method can bring a better sense of presence to the listener.

[0075] In addition, for example, the sound signal processing method related to the 7th technical solution of the present disclosure is the sound signal processing method related to the 6th technical solution, and in the above-mentioned generation step, the above-mentioned pneumatic sound signal is generated in a manner such that the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, and the volume of the above-mentioned pneumatic sound is increased as the calculated distance is smaller, and the volume of the above-mentioned pneumatic sound is increased as the calculated wind volume is greater.

[0076] Thus, the aerodynamic sound heard by the listener can be controlled according to the moving speed of the object, the distance between the object and the listener, and the wind volume, so the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0077] In addition, for example, the sound signal processing method of the 8th technical solution of the present disclosure is the sound signal processing method of any one of the 1st to 7th technical solutions, and in the above-mentioned generation step, the above-mentioned aerodynamic sound signal is generated based on the calculated above-mentioned moving speed, the calculated above-mentioned distance and the shape of the human ear.

[0078] Thus, the aerodynamic sound signal is generated according to the sound generation model based on the shape of the human ear, so the listener can hear the aerodynamic sound with a better sense of presence. In other words, the acoustic signal processing method can give the listener a better sense of presence.

[0079] In addition, for example, the sound signal processing method related to the 9th technical solution of the present disclosure is an sound signal processing method related to any one of the 1st to 7th technical solutions, including a storage step of storing aerodynamic sound data, wherein the aerodynamic sound data is data obtained by collecting sounds produced by wind reaching human ears or a model simulating human ears; in the above-mentioned generation step, the above-mentioned aerodynamic sound signal is generated based on the above-mentioned stored aerodynamic sound data, the above-mentioned calculated moving speed and the above-mentioned calculated distance.

[0080] Thus, the aerodynamic sound signal is generated based on the data collected from the actually generated sound, so the listener can hear the aerodynamic sound with a better sense of presence. In other words, the acoustic signal processing method can give the listener a better sense of presence.

[0081] In addition, for example, the sound signal processing method related to the 10th technical solution of the present disclosure is the sound signal processing method related to the 6th or 7th technical solution, and in the above-mentioned third calculation step, the above-mentioned wind volume is calculated based on the area of ​​the above-mentioned object observed from the moving direction of the above-mentioned object represented by the above-mentioned shape information obtained and the above-mentioned moving speed calculated.

[0082] Thus, the air volume is calculated based on the area and moving speed of the object as viewed from the moving direction of the object indicated by the shape information. Furthermore, an aerodynamic sound signal is generated based on the air volume, the moving speed of the object, and the distance between the object and the listener, so that the listener can hear aerodynamic sound with a better sense of presence. That is, the acoustic signal processing method can give the listener a better sense of presence.

[0083] In addition, for example, the sound signal processing method related to the 11th technical solution of the present disclosure is the sound signal processing method related to the 6th or 7th technical solution, and in the above-mentioned first calculation step, the above-mentioned moving speed and moving direction of the above-mentioned object are calculated based on the above-mentioned first position information obtained; in the above-mentioned third calculation step, the above-mentioned wind volume is calculated based on the above-mentioned shape information obtained, the calculated above-mentioned moving direction and the calculated above-mentioned moving speed.

[0084] Thus, the air volume is calculated based on the shape information, the moving direction, and the moving speed. Furthermore, an aerodynamic sound signal is generated based on the air volume, the moving speed of the object, and the distance between the object and the listener, so that the listener can hear aerodynamic sound with a better sense of presence. That is, the sound signal processing method can bring a better sense of presence to the listener.

[0085] In addition, the sound signal processing method related to the 12th technical solution of the present disclosure includes: a generation step, processing the noise signal separately with multiple frequency band emphasis filters, synthesizing the multiple processed noise signals, thereby generating an aerodynamic sound signal, which represents the aerodynamic sound generated by the wind reaching the ears of the listener in the virtual space; and an output step, outputting the generated aerodynamic sound signal, the multiple frequency band emphasis filters are respectively used to simulate the aerodynamic sound generated by the shape of the ears or head of the listener.

[0086] Thus, the sound signal processing method can generate an aerodynamic sound signal representing the aerodynamic sound generated by the wind reaching the ears of the listener in the virtual space for the listener in the virtual space, and output the generated aerodynamic sound signal. The aerodynamic sound signal is output to headphones or speakers, etc., and the listener can hear the aerodynamic sound. Therefore, the listener can feel that the wind is blowing in the virtual space and can get a sense of presence. That is, the sound signal processing method related to a technical solution of the present disclosure can give the listener a sense of presence.

[0087] In addition, for example, the sound signal processing method related to the 13th technical solution of the present disclosure is the sound signal processing method related to the 12th technical solution, and the above-mentioned multiple frequency band emphasis filters multiply the noise signal processed by the frequency band emphasis filter among the above-mentioned multiple noise signals after being processed and the prescribed coefficient corresponding to the frequency band emphasis filter, and in the above-mentioned generation step, the above-mentioned synthesis is performed after the above-mentioned multiplication.

[0088] Thus, the aerodynamic sound generated by the acoustic signal processing method becomes a sound that is closer to the aerodynamic sound heard by the listener in the real space. Therefore, the listener can get a better sense of presence. That is, the acoustic signal processing method of a technical solution of the present disclosure can bring a better sense of presence to the listener.

[0089] In addition, for example, the sound signal processing method according to the 14th technical solution of the present disclosure is the sound signal processing method according to the 12th or 13th technical solution, and the characteristics of each of the plurality of band emphasis filters vary according to the wind speed of the wind reaching the ear.

[0090] Thus, the aerodynamic sound generated by the acoustic signal processing method becomes a sound that is closer to the aerodynamic sound heard by the listener in the real space. Therefore, the listener can get a better sense of presence. That is, the acoustic signal processing method of a technical solution of the present disclosure can bring a better sense of presence to the listener.

[0091] In addition, for example, the sound signal processing method according to the 15th technical solution of the present disclosure is the sound signal processing method according to any one of the 12th to 14th technical solutions, and the characteristics of each of the plurality of band emphasis filters vary according to the wind direction of the wind reaching the ear.

[0092] Thus, the aerodynamic sound generated by the acoustic signal processing method becomes a sound that is closer to the aerodynamic sound heard by the listener in the real space. Therefore, the listener can get a better sense of presence. That is, the acoustic signal processing method of a technical solution of the present disclosure can bring a better sense of presence to the listener.

[0093] Furthermore, a computer program according to a sixteenth aspect of the present disclosure is a program for causing a computer to execute the sound signal processing method according to any one of the first to sixteenth aspects.

[0094] Thereby, the computer can execute the above-mentioned sound signal processing method according to the computer program.

[0095] In addition, the sound signal processing device related to the 17th technical solution of the present disclosure comprises: an acquisition unit, which acquires first position information indicating the position of an object as a moving body in a virtual space, and second position information indicating the position of a listener in the above virtual space; a first calculation unit, which calculates the moving speed of the above object based on the acquired first position information; a second calculation unit, which calculates the distance between the above object and the above listener based on the acquired first position information and the acquired second position information; a generation unit, which generates an aerodynamic sound signal based on the calculated moving speed and the calculated distance, the aerodynamic sound signal indicating the aerodynamic sound generated by the wind generated by the movement of the above object reaching the ears of the above listener; and an output unit, which outputs the generated aerodynamic sound signal.

[0096] Thus, the sound signal processing device can generate an aerodynamic sound signal representing the aerodynamic sound generated by the wind generated by the movement of the object reaching the ears of the listener based on the position of the object in the virtual space and the position of the listener, and output the generated aerodynamic sound signal. The aerodynamic sound signal is output to headphones or speakers, etc., and the listener can hear the aerodynamic sound. Therefore, the listener can perceive that the object is moving and can get a sense of presence. That is, the sound signal processing device related to a technical solution of the present disclosure can give the listener a sense of presence.

[0097] Furthermore, these inclusive or specific technical solutions may also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0098] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0099] In addition, the embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, components, configuration positions and connection forms of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure.

[0100] In addition, in the following description, elements are sometimes given ordinals such as 1 and 2. These ordinals are given to elements for the purpose of identifying the elements and do not necessarily correspond to a meaningful order. These ordinals may be replaced appropriately, newly given, or removed.

[0101] In addition, each figure is a schematic diagram and does not necessarily illustrate the exact figure. Therefore, the scales and the like in each figure are not necessarily the same. In each figure, the same reference numerals are given to substantially the same structure, and repeated descriptions are omitted or simplified.

[0102] In this specification, terms such as vertical and other terms indicating the relationship between elements and numerical ranges do not express strict meanings but mean substantially equivalent ranges, for example, including differences of about several percentage points.

[0103] (Implementation Method)

[0104] [Examples of Devices to Which the Sound Processing Technology or Encoding / Decoding Technology of the Present Disclosure Can Be Applied]

[0105] (Stereo sound reproduction system)

[0106] Figure 2B 1 is a diagram showing an immersive audio reproduction system A0000 as an example of a system to which the audio processing or decoding processing of the present disclosure can be applied. The immersive audio reproduction system A0000 includes an audio signal processing device A0001 and an audio presentation device A0002.

[0107] The sound signal processing device A0001 performs sound processing on the sound signal emitted by the virtual sound source, and generates a sound signal after sound processing to prompt the listener (i.e., the receiver). The sound signal is not limited to the human voice, as long as it is an audible sound. For example, sound processing is a signal processing performed on the sound signal in order to reproduce one or more sound-related effects received from the time when the sound is emitted to the time when the listener hears it. The sound signal processing device A0001 performs sound processing based on information that records the causes of the above-mentioned sound-related effects. Spatial information includes, for example, information indicating the position of the sound source, the listener, and the surrounding objects, information indicating the shape of the space, parameters related to the propagation of sound, etc. The sound signal processing device A0001 is, for example, a PC (Personal Computer), a smart phone, a tablet computer, or a game console.

[0108] The sound-processed signal is prompted to the listener (user) from the sound prompting device A0002. The sound prompting device A0002 is connected to the sound signal processing device A0001 via wireless or wired communication. The sound signal generated by the sound signal processing device A0001 after sound processing is transmitted to the sound prompting device A0002 via wireless or wired communication. In the case where the sound prompting device A0002 is composed of multiple devices such as a device for the right ear and a device for the left ear, the multiple devices synchronously prompt the sound by communicating between the multiple devices or each of the multiple devices communicates with the sound signal processing device A0001. The sound prompting device A0002 is, for example, headphones, earplugs, a head-mounted display worn on the listener's head, or a surround speaker composed of multiple fixed speakers.

[0109] Furthermore, the stereo sound reproduction system A0000 may be used in combination with an image presentation device or a stereoscopic image presentation device that visually provides an ER (Extended Reality) experience including AR / VR.

[0110] in addition, Figure 2B The system configuration example shows that the sound signal processing device A0001 and the sound prompting device A0002 are different devices, but the stereo sound reproduction system to which the sound signal processing method or decoding method disclosed in the present invention can be applied is not limited to Figure 2B For example, the sound signal processing device A0001 may be included in the sound prompting device A0002, and the sound prompting device A0002 may perform both sound processing and sound prompting. In addition, the sound signal processing device A0001 and the sound prompting device A0002 may share the sound processing described in the present disclosure, or a server connected to the sound signal processing device A0001 or the sound prompting device A0002 via a network may implement part or all of the sound processing described in the present disclosure.

[0111] In addition, although the above description refers to the sound signal processing device A0001, when the sound signal processing device A0001 performs sound processing by decoding a bit stream generated by encoding data of at least a part of spatial information used in a sound signal or sound processing, the sound signal processing device A0001 may also be referred to as a decoding device.

[0112] (Example of encoding device)

[0113] Figure 2C It is a functional block diagram showing the structure of an encoding device A0100 which is an example of the encoding device of the present disclosure.

[0114] Input data A0101 is data to be encoded including spatial information and / or a sound signal input to encoder A0102. The details of spatial information will be described later.

[0115] The encoder A0102 encodes the input data A0101 to generate encoded data A0103. The encoded data A0103 is, for example, a bit stream generated by encoding processing.

[0116] The memory A0104 stores the encoded data A0103. The memory A0104 may be, for example, a hard disk or an SSD (Solid-State Drive), or may be other memory.

[0117] In addition, in the above description, a bit stream generated by encoding processing is cited as an example of the encoded data A0103 stored in the memory A0104, but it can also be data other than a bit stream. For example, the encoding device A0100 can also store the converted data generated by converting the bit stream into a specified data format in the memory A0104. The converted data can also be, for example, a file or a multiplexed stream storing one or more bit streams. Here, the file is a file having a file format such as ISOBMFF (ISO Base Media File Format). In addition, the encoded data A0103 can also be in the form of multiple packets generated by dividing the above-mentioned bit stream or file. In the case of converting the bit stream generated by the encoder A0102 into data different from the bit stream, the encoding device A0100 can also have a conversion unit not shown in the figure, and the conversion process can also be performed by a CPU (Central Processing Unit).

[0118] (Example of decoding device)

[0119] Figure 2D It is a functional block diagram showing the structure of a decoding device A0110 which is an example of a decoding device according to the present disclosure.

[0120] The memory A0114 stores, for example, the same data as the coded data A0103 generated by the coding device A0100. The memory A0114 reads the stored data and inputs it as input data A0113 of the decoder A0112. The input data A0113 is, for example, a bit stream to be decoded. The memory A0114 may be, for example, a hard disk or an SSD, or other memory.

[0121] In addition, the decoding device A0110 may use the transformed data generated by transforming the read data instead of the data stored in the memory A0114 as input data A0113. The data before the transformation may be, for example, multiplexed data storing one or more bit streams. Here, the multiplexed data may also be a file having a file format such as ISOBMFF. In addition, the data before the transformation may also be in the form of a plurality of packets generated by dividing the above-mentioned bit stream or file. In the case of transforming data different from the bit stream read from the memory A0114 into a bit stream, the decoding device A0110 may also include a transformation unit not shown in the figure, or the transformation process may be performed by the CPU.

[0122] Decoder A0112 decodes input data A0113 to generate a sound signal A0111 for presenting a prompt to the listener.

[0123] (Another example of an encoding device)

[0124] Figure 2E 1 is a functional block diagram showing the structure of an encoding device A0120 which is another example of the encoding device of the present disclosure. Figure 2E For Figure 2C The same function as the structure of Figure 2C The same reference numerals are used for the components, and description of these components is omitted.

[0125] The coding device A0100 stores the coded data A0103 in the memory A0104 , but the coding device A0120 is different from the coding device A0100 in that it includes a transmission unit A0121 that transmits the coded data A0103 to the outside.

[0126] The transmission unit A0121 transmits a transmission signal A0122 to another device or server based on the coded data A0103 or data in another data format generated by transforming the coded data A0103. The data used to generate the transmission signal A0122 is, for example, the bit stream, multiplexed data, file, or packet described in the coding device A0100.

[0127] (Another example of a decoding device)

[0128] Figure 2F 1 is a functional block diagram showing the structure of a decoding device A0130 which is another example of the decoding device of the present disclosure. Figure 2F For Figure 2D The same function as the structure of Figure 2D The same reference numerals are used for the components, and description of these components is omitted.

[0129] The decoding device A0110 reads the input data A0113 from the memory A0114 , whereas the decoding device A0130 is different from the decoding device A0110 in that it includes a receiving unit A0131 that receives the input data A0113 from the outside.

[0130] The receiving unit A0131 receives the received signal A0132 to obtain received data, and outputs input data A0113 to be input to the decoder A0112. The received data may be the same as the input data A0113 to be input to the decoder A0112, or may be data in a data format different from that of the input data A0113. When the received data is data in a data format different from that of the input data A0113, the receiving unit A0131 may convert the received data into the input data A0113, or a conversion unit or CPU (not shown) included in the decoding device A0130 may convert the received data into the input data A0113. The received data is, for example, the bit stream, multiplexed data, file, or packet described in the encoding device A0120.

[0131] (Decoder Functional Description)

[0132] Figure 2G It means as Figure 2D or Figure 2F A functional block diagram of the structure of a decoder A0200 which is an example of the decoder A0112 in FIG.

[0133] The input data A0113 is a coded bit stream, and includes coded audio data which is a coded audio signal and metadata used in audio processing.

[0134] The spatial information management unit A0201 obtains metadata included in the input data A0113 and parses the metadata. The metadata includes information describing elements that act on the sound and are configured in the sound space. The spatial information management unit A0201 manages the spatial information required for sound processing obtained by parsing the metadata, and provides the spatial information to the rendering unit A0203. In addition, in the present disclosure, the information used in the sound processing is referred to as spatial information, but it may be referred to by other names. Regarding the information used in the sound processing, for example, the information used in the sound processing may be referred to as sound spatial information, or it may be referred to as scene information. In addition, in the case where the information used in the sound processing changes over time, the spatial information input to the rendering unit A0203 may also be referred to as a spatial state, a sound spatial state, a scene state, or the like.

[0135] In addition, the spatial information can also be managed by sound space or by scene. For example, when different rooms are represented as virtual spaces, each room can be managed as a scene of a different sound space, or the spatial information can be managed as a scene that is different depending on the occasion of the representation despite being the same space. In the management of spatial information, an identifier for identifying each piece of spatial information can also be assigned. The data of the spatial information can be included in a bit stream as a form of input data, or the bit stream can include an identifier of the spatial information and the data of the spatial information can be obtained from outside the bit stream. In the case where only the identifier of the spatial information is included in the bit stream, the identifier of the spatial information can also be used during rendering to obtain the data of the spatial information stored in the memory of the sound signal processing device A0001 or in an external server as input data.

[0136] In addition, the information managed by the spatial information management unit A0201 is not limited to the information included in the bitstream. For example, the input data A0113 may include data representing spatial characteristics or structures obtained from a software application or server that provides VR or AR as data not included in the bitstream. In addition, for example, the input data A0113 may include data representing characteristics or positions of listeners or objects as data not included in the bitstream. In addition, the input data A0113 may include information obtained by a sensor provided by a terminal including a decoding device, or information representing the position of the terminal inferred based on the information obtained by the sensor, as information representing the position of the listener. That is, the spatial information management unit A0201 may also communicate with an external system or server to obtain spatial information and the position of the listener. In addition, the spatial information management unit A0201 may also obtain clock synchronization information from an external system and perform processing synchronized with the clock of the rendering unit A0203. In addition, the space in the above description may be a virtually formed space, i.e., a VR space, or a real space (real space) or a virtual space corresponding to the real space, i.e., an AR space or an MR (Mixed Reality) space. In addition, the virtual space may also be referred to as a sound field or a sound space. In addition, the information indicating the position in the above description may be information such as coordinate values ​​indicating the position in the space, information indicating the relative position relative to a predetermined reference position, or information indicating the movement or acceleration of the position in the space.

[0137] The sound data decoder A0202 decodes the encoded sound data contained in the input data A0113 to obtain a sound signal.

[0138] The coded audio data acquired by the stereo sound reproduction system A0000 is, for example, a bit stream coded in a prescribed format such as MPEG-H 3D Audio (ISO / IEC 23008-3). MPEG-H 3D Audio is merely an example of a coding method that can be used when generating coded audio data included in a bit stream, and may include a bit stream and coded audio data coded in other coding methods. For example, the coding method used may be non-reversible coding such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), Vorbis, or reversible coding such as ALAC (Apple Lossless Audio Codec), FLAC (Free Lossless Audio Codec), or any coding method other than the above may be used. For example, PCM (pulse code modulation) data may be one type of coded audio data. In this case, the decoding process may be a process of converting an N-bit binary number into a number format (eg, floating point format) that can be processed by the rendering unit A0203, for example, when the number of quantization bits of the PCM data is N.

[0139] The rendering unit A0203 takes the sound signal and spatial information as input, performs audio processing on the sound signal using the spatial information, and outputs the sound signal A0111 after the audio processing.

[0140] Before starting rendering, the spatial information management unit A0201 reads metadata of the input signal, detects rendering items such as objects or sounds specified by the spatial information, and sends them to the rendering unit A0203. After the rendering starts, the spatial information management unit A0201 grasps the changes in the spatial information and the listener's position over time, updates the spatial information and manages it. In addition, the spatial information management unit A0201 sends the updated spatial information to the rendering unit A0203. The rendering unit A0203 generates a sound signal with sound processing added based on the sound signal included in the input data and the spatial information received from the spatial information management unit A0201, and outputs it.

[0141] The spatial information update process and the sound signal output process with added sound processing may be executed by the same thread, or the spatial information management unit A0201 and the rendering unit A0203 may be assigned to separate threads. When the spatial information update process and the sound signal output process with added sound processing are processed by different threads, the thread startup frequency may be set separately, or the processes may be executed in parallel.

[0142] Since the spatial information management unit A0201 and the rendering unit A0203 execute processing in different independent threads, the computing resources can be preferentially allocated to the rendering unit A0203. Therefore, in the case of sound processing in which no delay can be allowed, for example, even in the case of a 1-sample (0.02msec) delay, a small noise can be generated. The sound processing can be safely implemented. At this time, the allocation of computing resources to the spatial information management unit A0201 is restricted. However, the update of spatial information is a lower-frequency process than the output processing of the sound signal (for example, a process such as updating the orientation of the listener's face). Therefore, unlike the output processing of the sound signal, it does not require an instantaneous response, so even if the allocation of computing resources is restricted, it will not have a significant impact on the sound quality brought to the listener.

[0143] The updating of spatial information can be performed periodically at a preset time or period, or when a preset condition is met. In addition, the updating of spatial information can be performed manually by the listener or the manager of the sound space, or it can be performed in response to changes in the external system. For example, when the listener operates the controller and the standing position of his or her avatar is instantly tilted, or the time is instantly moved forward or backward, or when the manager of the virtual space suddenly changes the scene, the thread configured with the spatial information management unit A0201 can be started as a one-time interrupt processing in addition to the regular startup.

[0144] The information update thread that performs the update processing of spatial information is responsible for, for example, updating the position or orientation of the listener's avatar configured in the virtual space based on the position or orientation of the VR goggles worn by the listener, and updating the position of objects moving in the virtual space, etc., provided in a processing thread that is started at a relatively low frequency of about tens of Hz. Such a processing thread with a low occurrence frequency can also be used to perform processing that reflects the properties of direct sound. This is because the frequency of changes in the properties of direct sound is lower than the frequency of occurrence of audio processing frames used for audio output. On the contrary, doing so can relatively reduce the computational load of the processing, and if the information is updated at an unnecessarily fast frequency, there is a risk of generating pulsed noise, so this risk can also be avoided.

[0145] Figure 2H It means as Figure 2D or Figure 2F A functional block diagram of the structure of a decoder A0210 which is another example of the decoder A0112 in FIG.

[0146] Figure 2H The input data A0113 does not include coded audio data but includes an uncoded audio signal. Figure 2G The input data A0113 includes a bit stream including metadata and an audio signal.

[0147] Space Information Management Department A0211 and Figure 2G The spatial information management unit A0201 is the same as that of the spatial information management unit A0201, so the description is omitted.

[0148] Rendering Department A0213 and Figure 2G The rendering part A0203 is the same, so the description is omitted.

[0149] In addition, in the above description Figure 2H The structure of is called a decoder, but it can also be called an audio processing unit that performs audio processing. In addition, a device including an audio processing unit can also be called an audio processing device instead of a decoding device. In addition, the audio signal processing device A0001 can also be called an audio processing device.

[0150] (Physical Structure of Sound Signal Processing Device)

[0151] Fig.2I is a diagram showing an example of the physical structure of an audio signal processing device. Fig.2I The audio signal processing device may also be a decoding device. In addition, a part of the structure described here may also be equipped in the sound prompting device A0002. In addition, Fig.2I The illustrated sound signal processing device is an example of the sound signal processing device A0001 described above.

[0152] Fig.2I The sound signal processing device includes a processor, a memory, a communication IF, a sensor, and a speaker.

[0153] The processor is, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit), and the CPU, DSP or GPU may execute a program stored in a memory to implement the audio processing or decoding processing of the present invention. In addition, the processor may also be a dedicated circuit for performing signal processing of sound signals including the audio processing of the present invention.

[0154] The memory is composed of, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). In addition, the memory may also include an internal memory embedded in a CPU or a GPU.

[0155] The communication IF (Interface) is a communication module corresponding to a communication method such as Bluetooth (registered trademark) or WIGI (registered trademark). Fig.2I The acoustic signal processing device shown has a function of communicating with other communication devices via a communication IF, and acquires a bit stream to be decoded, and stores the acquired bit stream in, for example, a memory.

[0156] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is cited as an example of the communication method, but it can also correspond to communication methods such as LTE (Long Term Evolution), NR (New Radio) or Wi-Fi (registered trademark). In addition, the communication IF may not be a wireless communication method as described above, but a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), etc.

[0157] The sensor performs sensing for estimating the position or orientation of the listener. Specifically, the sensor estimates the position and / or orientation of the listener based on one or more detection results of the position, orientation, movement, speed, angular velocity, or acceleration of a part or the whole of the body such as the head of the listener, and generates position information indicating the position and / or orientation of the listener. In addition, the position information may also be information indicating the position and / or orientation of the listener in real space, or information indicating the displacement of the position and / or orientation of the listener based on the position and / or orientation of the listener at a specified point in time. In addition, the position information may also be information indicating the relative position and / or orientation to an external device having a stereo sound reproduction system or a sensor.

[0158] The sensor may be, for example, a camera or other imaging device or a distance measuring device such as LiDAR (Light Detection And Ranging), and may capture the movement of the listener's head and detect the movement of the listener's head by processing the captured image. In addition, as a sensor, for example, a device that uses wireless position estimation using any frequency band such as millimeter waves may be used.

[0159] in addition, Fig.2I The acoustic signal processing device shown in FIG. 1 may also obtain position information from an external device having a sensor via a communication IF. In this case, the acoustic signal processing device may not include a sensor. Here, the external device is, for example, Figure 2B The sound presentation device A0002 described in , or a stereoscopic image reproduction device worn on the head of the listener, etc. In this case, the sensor is composed of a combination of various sensors such as a gyro sensor and an acceleration sensor.

[0160] For example, as the speed of movement of the listener's head, the sensor can detect the angular velocity of rotation about at least one of the three axes orthogonal to each other in the sound space, or the acceleration of displacement in the direction of displacement about at least one of the three axes.

[0161] For example, as the amount of movement of the listener's head, the sensor can detect the amount of rotation with at least one of the three axes orthogonal to each other in the sound space as the rotation axis, and can also detect the amount of displacement with at least one of the above three axes as the displacement direction. Specifically, the sensor detects 6DoF (position (x, y, z) and angle (yaw, pitch, roll) as the position of the listener. The sensor is composed of a combination of various sensors for detecting movement, such as gyroscope sensors and acceleration sensors.

[0162] In addition, the sensor only needs to detect the position of the listener, and can also be implemented by a camera or a GPS (Global Positioning System) receiver, etc. It is also possible to use position information obtained by self-position estimation using LiDAR (Laser Imaging Detection and Ranging), etc. For example, when the sound signal reproduction system is implemented by a smartphone, the sensor is built into the smartphone.

[0163] In addition, sensors can also include detection Fig.2I A temperature sensor such as a thermocouple for detecting the temperature of the sound signal processing device shown, and a sensor for detecting the remaining amount of a battery included in or connected to the sound signal processing device, etc.

[0164] The speaker has a driving mechanism such as a diaphragm, a magnet or a voice coil, and an amplifier, and presents the sound signal after acoustic processing to the listener as sound. The speaker operates the driving mechanism according to the sound signal amplified by the amplifier (more specifically, a waveform signal representing the waveform of the sound), and the driving mechanism vibrates the diaphragm. In this way, the diaphragm vibrating in response to the sound signal generates sound waves, which propagate in the air and are transmitted to the listener's ears, and the listener perceives the sound.

[0165] In addition, here are Fig.2I The example of the case where the audio signal processing device shown in the figure has a speaker and outputs the audio signal after the audio processing through the speaker is described, but the audio signal prompting mechanism is not limited to the above-mentioned structure. For example, the audio signal after the audio processing can also be output to the external audio prompting device A0002 connected by the communication module. The communication performed by the communication module can be either wired or wireless. In addition, as another example, it can also be Fig.2I The audio signal processing device shown has a terminal for outputting an analog signal of sound, and a cable of an earplug or the like is connected to the terminal to prompt a sound signal from the earplug or the like. In the above case, the earphone, earplug, head-mounted display, neck speaker, wearable speaker, or surround speaker composed of a plurality of fixed speakers worn on the head or a part of the body of the listener as the sound prompting device A0002 reproduces the sound signal.

[0166] (Physical Structure of Encoding Device)

[0167] Figure 2J is a diagram showing an example of the physical structure of an encoding device. Figure 2J The illustrated encoding device is an example of the encoding devices A0100 and A0120 described above.

[0168] Figure 2J The encoding device includes a processor, a memory and a communication IF.

[0169] The processor is, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and the encoding process of the present disclosure may be implemented by executing a program stored in a memory by the CPU or GPU. In addition, the processor may also be a dedicated circuit for performing signal processing on a sound signal including the encoding process of the present disclosure.

[0170] The memory is composed of, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). In addition, the memory may also include an internal memory embedded in a CPU or a GPU.

[0171] The communication IF (Interface) is a communication module corresponding to a communication method such as Bluetooth (registered trademark) or WIGI (registered trademark). The encoding device has a function of communicating with other communication devices via the communication IF, and transmits an encoded bit stream.

[0172] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is cited as the communication method, but it can also correspond to communication methods such as LTE (LongTerm Evolution), NR (New Radio) or Wi-Fi (registered trademark). In addition, the communication IF may not be a wireless communication method as described above, but a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), etc.

[0173] [constitute]

[0174] Next, the configuration of the sound signal processing device 100 according to the embodiment will be described. Figure 3A : is a block diagram showing the functional configuration of the sound signal processing device 100 according to the present embodiment.

[0175] The sound signal processing device 100 according to the present embodiment is a device for generating and outputting an aerodynamic sound signal, which represents the aerodynamic sound generated by the wind W generated by the movement of the object in the virtual space (sound reproduction space) reaching the ears of the listener L. In addition, the aerodynamic sound is also the sound generated by the wind W in the virtual space reaching the ears of the listener L, and is also the sound generated by the shape of the ears or head of the listener L. As an example, the sound signal processing device 100 according to the present embodiment is a device applied to various applications in virtual spaces such as virtual reality or augmented reality (VR / AR).

[0176] Objects in the virtual space are not particularly limited as long as they are objects included in the content displayed on the display unit 300 that displays the content executed in the virtual space (here, images as an example). The object is a moving body, and includes, for example, objects representing animals, plants, artificial objects, natural objects, and the like. Examples of objects representing artificial objects include vehicles, bicycles, and airplanes. In addition, examples of objects representing artificial objects include sporting goods such as baseball bats and tennis rackets; furniture such as tables, chairs, electric fans, and clocks; and buildings such as residential complexes and commercial facilities. In addition, as an example, an object may be at least one of an object that can move within the content and an object that is moved, but is not limited to this. In addition, Figure 2A The electric fan 600 shown is installed on the ceiling, and although the electric fan 600 itself does not move, the blades 601 of the electric fan 600 move. Such an electric fan 600 is also included in this object.

[0177] In addition, the aerodynamic sound related to the present embodiment is the sound generated by the wind W generated by the movement of the object in the virtual space and reaches the ear of the listener L. More specifically, the aerodynamic sound is the sound caused by the wind W generated by the movement of the air caused by the movement of the object. In addition, the ear refers to at least one of the concha and the outer ear.

[0178] The acoustic signal processing device 100 generates an aerodynamic sound signal representing aerodynamic sound in a virtual space, and outputs the signal to the earphone 200 .

[0179] Next, the earphone 200 will be described.

[0180] The earphone 200 is a device for reproducing aerodynamic sound and is a sound output device. More specifically, the earphone 200 reproduces aerodynamic sound based on the aerodynamic sound signal output by the sound signal processing device 100. Thus, the listener L can hear the aerodynamic sound. In addition to the earphone 200, other output channels such as speakers may be used.

[0181] like Figure 3A As shown, the headset 200 includes a head sensor unit 201 and an output unit 202 .

[0182] The head sensor unit 201 senses the position of the listener L in the virtual space, which is determined by the horizontal coordinates and the vertical height, and outputs second position information indicating the position of the listener L in the virtual space to the sound signal processing device 100 .

[0183] The head sensor unit 201 may sense 6DoF (Degrees of Freedom) information of the head of the listener L. For example, the head sensor unit 201 may be an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetic sensor, or a combination thereof.

[0184] The output unit 202 is a device that reproduces the sound reaching the listener L in the sound reproduction space. More specifically, the output unit 202 reproduces the aerodynamic sound based on the aerodynamic sound signal representing the aerodynamic sound processed by the acoustic signal processing device 100 and output from the acoustic signal processing device 100 .

[0185] Next, the display unit 300 will be described.

[0186] The display unit 300 is a display device that displays content (images) including objects in a virtual space. The processing for the display unit 300 to display content will be described later. The display unit 300 is implemented by a display panel such as a liquid crystal panel or an organic EL (ElectroLuminescence) panel.

[0187] Furthermore, Figure 3A The audio signal processing device 100 shown will be described.

[0188] like Figure 3A As shown, the sound signal processing device 100 includes an acquisition unit 110 , a calculation unit 120 , a generation unit 130 , an output unit 140 , and a storage unit 150 .

[0189] The acquisition unit 110 acquires the first position information and the second position information from, for example, an input signal. The input signal is described below. The first position information is information indicating the position of an object in a virtual space. The second position information is information indicating the position of a listener L in a virtual space, as described above. In addition, the acquisition unit 110 may also acquire the first position information and the second position information from sources other than the input signal.

[0190] Furthermore, the acquisition unit 110 may also acquire the availability information and the shape information. The availability information is information indicating whether the object generates aerodynamic sound. The shape information is information indicating the shape of the object.

[0191] In this embodiment, the acquisition unit 110 acquires the first position information, the capability information, and the shape information from a device other than the sound signal processing device 100 (here, for example, the server device 500 such as a cloud server), but the present invention is not limited thereto. In addition, the acquisition unit 110 acquires the second position information from the earphone 200 (more specifically, the head sensor unit 201), but the present invention is not limited thereto.

[0192] The input signal is composed of, for example, spatial information, sensor information, and sound data (sound signal). In addition, the above information and sound data may be included in one input signal, or in a plurality of different signals. The input signal may also include a bit stream composed of sound data and metadata (control information), in which case the metadata may also include spatial information and information for identifying sound data.

[0193] The first position information, the second position information, the shape information, and the availability information described above may be included in the input signal or may be included in other information. More specifically, the first information, the shape information, and the availability information may be included in the spatial information, and the second information may be generated based on information obtained from the sensor information. The sensor information may be obtained from the head sensor unit 201 or from other external devices.

[0194] Spatial information is information about the sound space (three-dimensional sound field) formed by the stereo sound reproduction system, and is composed of information about objects contained in the sound space and information about the listener. Among the objects, there are sound source objects that emit sound and become sound sources, and non-sound objects that do not emit sound. Non-sound objects function as obstacle objects that reflect the sound emitted by sound source objects, but there are also cases where they function as obstacle objects that reflect the sound emitted by other sound source objects as sound source objects. Obstacle objects can also be called reflection objects.

[0195] The information given to both the sound source object and the non-sound emitting object includes position information, shape information, and a volume attenuation rate when the object reflects sound.

[0196] The position information is represented by the coordinate values ​​of three axes, such as the X-axis, Y-axis, and Z-axis in Euclidean space, but it does not necessarily have to be three-dimensional information. The position information may also be two-dimensional information represented by the coordinate values ​​of two axes, such as the X-axis and the Y-axis. The position information of the object is determined by the representative position of the shape represented by the grid or voxel.

[0197] The shape information may also include information about the material of the surface.

[0198] The attenuation rate can be expressed by a real number below 1 or above 0, or by a negative decibel value. Since the volume is not amplified by reflection in real space, the attenuation rate is set to a negative decibel value, but for example, in order to express the horror of an unreal space, an attenuation rate of 1 or above, that is, a positive decibel value, can be deliberately set. In addition, the attenuation rate can be set to a different value for each frequency band constituting a plurality of frequency bands, or a value can be set independently for each frequency band. In addition, when the attenuation rate is set for each type of material on the surface of the object, the corresponding attenuation rate value can also be used based on information related to the material of the surface.

[0199] In addition, the information assigned to the sound source object and the non-sound-generating object may also include information indicating whether the object is a living thing, or information indicating whether the object is a moving object, etc. In the case where the object is a moving object, the position information may also move over time, and the changed position information or the amount of change is transmitted to the rendering units A0203 and A0213.

[0200] The information related to the sound source object includes, in addition to the above-mentioned information commonly assigned to the sound source object and the non-sound-generating object, sound data and information required to radiate the sound data into the sound space. Sound data is data that represents the sound perceived by the listener, including information related to the frequency and strength of the sound. The sound data is typically a PCM signal, but it can also be data compressed using an encoding method such as MP3. In this case, it is necessary to decode the signal at least before it reaches the generation unit 130, so a decoding unit not shown in the figure may also be included in the rendering units A0203 and A0213. Alternatively, the signal may also be decoded by the sound data decoder A0202.

[0201] It is sufficient to set at least one sound data for one sound source object, or multiple sound data may be set. In addition, identification information for identifying each sound data may be given, and the identification information of the sound data may be stored as metadata as information related to the sound source object.

[0202] The information required for radiating sound data into the sound space may include, for example, information on a reference volume used as a reference when reproducing sound data, information related to the position of the sound source object, information related to the orientation of the sound source object, and information related to the directionality of the sound emitted by the sound source object.

[0203] The reference volume information is, for example, the effective value of the amplitude value of the sound data at the sound source position when the sound data is radiated into the sound space, and may also be expressed as a decibel (db) value with a floating decimal point. For example, when the reference volume is 0db, the reference volume information may also indicate that the sound is radiated to the sound space from the position indicated by the information related to the above position without increasing or decreasing the volume of the signal level represented by the sound data. In the case of -6db, the reference volume information may also indicate that the sound is radiated to the sound space from the position indicated by the information related to the above position with the volume of the signal level represented by the sound data reduced to approximately half. The reference volume information may also be assigned to one sound data or to a plurality of sound data together.

[0204] The volume information included in the information required to radiate the sound data into the sound space may include, for example, information indicating the time series changes of the volume of the sound source. For example, when the sound space is a virtual conference room and the sound source is a speaker, the volume changes intermittently in a short period of time. If it is expressed more simply, it can be said that the sound part and the silent part are produced alternately. In addition, when the sound space is a concert hall and the sound source is a performer, the volume is maintained for a certain length of time. In addition, when the sound space is a battlefield and the sound source is an explosive, the volume of the explosion only increases for a moment, and then it is silent and continues. In this way, the information on the volume of the sound source includes not only the information on the size of the sound, but also the information on the changes in the size of the sound, and such information can also be set as information indicating the nature of the sound data.

[0205] Here, the information on the change in the volume of the sound may also be data representing the frequency characteristics in a time series. The information on the change in the volume of the sound may also be data representing the duration of the sound interval. The information on the change in the volume of the sound may also be data representing the duration of the sound interval and the time length of the silent interval. The information on the change in the volume of the sound may also be data listing the duration of the amplitude of the sound signal as constant (regarded as approximately constant) and the amplitude value of the signal during the duration in a plurality of time series. The information on the change in the volume of the sound may also be data listing the duration of the frequency characteristics of the sound signal as constant and the frequency characteristics during the duration in a plurality of time series. The information on the change in the volume of the sound may also be data representing the approximate shape of a spectrogram as data. In addition, the volume as a reference for the above-mentioned frequency characteristics may also be set as the above-mentioned reference volume. The information indicating the reference volume and the information indicating the properties of the sound data are used not only for calculating the volume of the direct sound or the reflected sound to be perceived by the listener, but also for selection processing for selecting whether to make the listener perceive the sound.

[0206] The information related to the orientation is typically expressed by yaw, pitch, and roll. Alternatively, the rotation of roll may be omitted and expressed by azimuth (yaw) and elevation (pitch). The orientation information may also change over time and, if changed, is transmitted to the rendering units A0203 and A0213.

[0207] Information related to the listener is information related to the position information and orientation of the listener in the sound space. The position information is represented by the position of the XYZ axis of the Euclidean space, but it does not necessarily have to be three-dimensional information, and it can also be two-dimensional information. Information related to the orientation is typically represented by yaw, pitch, and roll. Alternatively, the rotation of roll can be omitted and represented by azimuth (yaw) and elevation (pitch). The position information and orientation information can also change over time, and when they change, they are transmitted to the rendering units A0203 and A0213.

[0208] The sensor information is information including the amount of rotation or displacement detected by the sensor worn by the listener and the position and orientation of the listener. The sensor information is transmitted to the rendering units A0203 and A0213, and the rendering units A0203 and A0213 update the information of the position and orientation of the listener based on the sensor information. For example, the sensor information can also use the position information obtained by the portable terminal using GPS, camera or LiDAR (Laser Imaging Detection and Ranging) to estimate its own position. In addition, information obtained from the outside via the communication module other than the sensor can also be detected as sensor information. Information indicating the temperature of the sound signal processing device 100 and information indicating the remaining battery level can also be obtained from the sensor as sensor information. Information indicating the computing resources (CPU capacity, memory resources, PC performance) of the sound signal processing device 100 or the sound prompt device A0002 can also be obtained in real time as sensor information.

[0209] Here, the first position information, the second position information, the feasibility information, and the shape information are described in more detail.

[0210] First, the first position information will be described.

[0211] As described above, the object in the virtual space is included in the content (image) displayed by the display unit 300, and may be at least one of an object that can be moved within the content and an object that can be moved. Figure 1 Vehicle 400 is shown.

[0212] The first position information is information indicating where the vehicle 400 in the virtual space is located at a certain point in time. In addition, in the virtual space, the vehicle 400 may move by driving by the driver. Therefore, the acquisition unit 110 continuously acquires the first position information. The acquisition unit 110 acquires the first position information every time, for example, the spatial information management units A0201 and A0211 execute an update of the spatial information.

[0213] Next, the second position information will be described.

[0214] The listener L can move in the virtual space. The second position information is information indicating where the listener L in the virtual space is at a certain point in time. In addition, since the listener L can move in the virtual space, the acquisition unit 110 continuously acquires the second position information. The acquisition unit 110 acquires the second position information every time the spatial information management units A0201 and A0211 execute an update of the spatial information, for example.

[0215] Next, shape information will be described.

[0216] Shape information is information indicating the shape of an object (e.g., vehicle 400) in a virtual space. Shape information indicates the shape of an object, and more specifically, indicates a three-dimensional shape as a rigid body of the object. The shape of the object is represented by, for example, a sphere, a cuboid, a cube, a polyhedron, a cone, a pyramid, a cylinder, a prism, or a combination thereof. In addition, shape information may be represented, for example, as mesh data, or as a set of multiple faces composed of voxels, a three-dimensional point group, or vertices having three-dimensional coordinates.

[0217] Next, the feasibility information will be described.

[0218] The ability information is information indicating whether the object (e.g., vehicle 400) generates a pneumatic sound. That is, the ability information indicates whether the pneumatic sound is generated or not. For example, when the ability information indicates that the pneumatic sound is generated, "1" is indicated as a flag, and when the ability information indicates that the pneumatic sound is not generated, "0" is indicated as a flag. The ability information can also be said to be flag information, which is an example of metadata related to the object.

[0219] In the enable / disable information, whether the object generates aerodynamic sound is predetermined according to the object represented by the enable / disable information. For example, whether the enable / disable information indicates true or false is predetermined by the administrator of the content (image) displayed on the display unit 300. In addition, the sound signal processing device 100 may include Figure 3A The selection unit (not shown) may be used to select whether or not to generate aerodynamic sound based on the availability information. In addition, the selection process may be performed by the selection unit in consideration of the overall computational load of the acoustic signal processing device 100 related to the generation of other sounds other than aerodynamic sound. In addition, the priority may be preset in the aerodynamic sound and the information of the priority may be included in the availability information. For example, the aerodynamic sound with a low priority may be selected not to be generated in consideration of the processing capacity of the acoustic signal processing device 100 or the overall computational load related to the generation of the sound space. In addition, the processing capacity of the acoustic signal processing device 100 may be compared with the overall computational load, and when the processing capacity of the acoustic signal processing device 100 is not sufficient, the aerodynamic sound may be selected not to be generated even if the availability information is true. In addition, when both the sound source information other than aerodynamic sound and the availability information indicating true are given to the object, when the processing capacity is not sufficient, the generation of the sound source may be prioritized and the generation of the aerodynamic sound may not be selected, and the availability information may include information pre-set as to which of the generation of the sound source and the generation of the aerodynamic sound is prioritized.

[0220] Here, in Figure 3B 2 shows an example (action example) of the process flow of the selection unit. Figure 3B1 is a flowchart of an operation example of the selection unit of the present embodiment. First, the selection unit analyzes the input signal and determines whether the input signal contains the ability information (S100). In the case where the processing capability or the computational load of the sound signal processing device 100 is considered in the selection of whether to generate the pneumatic sound, such information may be obtained in this step S100. Next, the selection unit determines whether to select the generation of the pneumatic sound (pneumatic sound selection) based on the ability information and the processing capability or the computational load of the sound signal processing device (S110).

[0221] Furthermore, the selection unit may perform not only the selection process of the pneumatic sound but also the selection process of other sounds.

[0222] The first position information includes identification information for identifying the object, the availability information also includes identification information for identifying the object, and the shape information also includes identification information for identifying the object.

[0223] Therefore, even if the acquisition unit 110 acquires the first position information, the capability information, and the shape information separately, the objects represented by the first position information, the capability information, and the shape information are identified by referring to the identification information contained in each of the first position information, the capability information, and the shape information. For example, it can be easily identified here that the objects represented by each of the first position information, the capability information, and the shape information are the same vehicle 400. That is, with respect to each of the first position information, the capability information, and the shape information acquired by the acquisition unit 110, by referring to the three pieces of identification information, it is clear that the first position information, the capability information, and the shape information are information related to the vehicle 400. Therefore, the first position information, the capability information, and the shape information are associated as information representing the vehicle 400.

[0224] In addition, the first position information, the availability information, the shape information, etc. mentioned above may also be included in the metadata, control information, or header information included in the input signal. That is, the sound signal processing device 100 (more specifically, the acquisition unit 110) may also acquire the metadata, control information, or header information included in the input signal, and perform sound processing based on the metadata, control information, or header information. In addition, the sound signal processing device 100 (more specifically, the acquisition unit 110) only needs to acquire the first position information, availability information, and shape information, and the acquisition location is not limited to the input signal.

[0225] In addition, the audio content information (sound signal) included in the input signal can be subjected to encoding processing such as MPEG-H 3D Audio (ISO / IEC 23008-3) (hereinafter referred to as MPEG-H 3D Audio). In addition, the technology used in the encoding processing is not limited to MPEG-H 3D Audio, and other well-known technologies can also be used. In addition, information other than the sound signal, such as the first position information, the availability information, and the shape information, can also be used as the encoding processing object. That is, the sound signal processing device 100 obtains the sound signal and metadata included in the encoded bit stream. In the sound signal processing device 100, the audio content information is obtained and decoded. In this embodiment, the sound signal processing device 100 functions as a decoder possessed by the decoding device, and more specifically, functions as the rendering units A0203 and A0213 possessed by the decoder. In addition, the terms such as audio content information or sound data in the present disclosure can be interpreted as replacing the sound signal itself, or information including the shape information or the availability information in terms of technical content.

[0226] Furthermore, the second position information may also be coded. That is, the acquisition unit 110 acquires the second position information and decodes it.

[0227] The acquisition unit 110 outputs the acquired first position information and second position information to the calculation unit 120. In addition, when the acquisition unit 110 acquires the availability information, the acquisition unit 110 outputs the acquired availability information to the generation unit 130 or the selection unit (not shown). In addition, when the acquisition unit 110 acquires the shape information, the acquisition unit 110 outputs the acquired shape information to the calculation unit 120 and the generation unit 130.

[0228] The calculation unit 120 calculates the moving speed of the object, the distance between the object and the listener L, the wind volume per unit time generated when the object moves, etc. The calculation unit 120 includes a first calculation unit 121 , a second calculation unit 122 , and a third calculation unit 123 .

[0229] The first calculation unit 121 calculates the moving speed of the object in the virtual space based on the first position information acquired by the acquisition unit 110. As described above, the acquisition unit 110 acquires the first position information indicating the position of the object in the virtual space each time the spatial information is updated by the spatial information management units A0201 and A0211. The first calculation unit 121 calculates the moving speed of the object in the virtual space by differentiating the position of the object with time in the virtual space based on the first position information (more specifically, a plurality of first position information) acquired each time the spatial information is updated by the spatial information management units A0201 and A0211. The first calculation unit 121 outputs the calculated moving speed to the generation unit 130.

[0230] Furthermore, the second calculation unit 122 calculates the distance between the object and the listener L based on the first position information and the second position information acquired by the acquisition unit 110. As described above, the acquisition unit 110 acquires the first position information and the second position information in the virtual space each time the spatial information is updated by the spatial information management units A0201 and A0211. The second calculation unit 122 calculates the distance between the object and the listener L in the virtual space based on the plurality of first position information and the plurality of second position information acquired each time the spatial information is updated by the spatial information management units A0201 and A0211. The second calculation unit 122 outputs the calculated distance between the object and the listener L to the generation unit 130.

[0231] Furthermore, when the acquisition unit 110 acquires the shape information, the third calculation unit 123 calculates the air volume per unit time generated when the object moves based on the moving speed calculated by the first calculation unit 121 and the shape information acquired by the acquisition unit 110 .

[0232] The generator 130 generates an aerodynamic sound signal indicating aerodynamic sound generated by the wind W generated by the movement of the object and reaching the ear of the listener L, based on the moving speed calculated by the first calculator 121 and the distance calculated by the second calculator 122. The generator 130 outputs the generated aerodynamic sound signal to the output unit 140.

[0233] The output unit 140 outputs the pneumatic sound signal generated by the generating unit 130. Here, the output unit 140 outputs the pneumatic sound signal to the earphone 200. Thus, the earphone 200 can reproduce the pneumatic sound indicated by the outputted pneumatic sound signal.

[0234] The storage unit 150 is a storage device that stores computer programs and the like executed by the acquisition unit 110 , the calculation unit 120 , the generation unit 130 , and the output unit 140 .

[0235] Here, the shape information related to this embodiment is described again. The shape information is information used to generate an image of an object in a virtual space, and is also information representing the shape of the object (vehicle 400). That is, the shape information is also information used to generate content (image) displayed on the display unit 300.

[0236] The acquisition unit 110 also outputs the acquired shape information to the display unit 300. The display unit 300 acquires the shape information output by the acquisition unit 110. The display unit 300 also acquires attribute information indicating attributes (such as color) other than the shape of the object (vehicle 400) in the virtual space. The display unit 300 may directly acquire the attribute information from a device (server device 500) other than the sound signal processing device 100, or may acquire the attribute information from the sound signal processing device 100. The display unit 300 generates and displays content (image) based on the acquired shape information and attribute information.

[0237] Hereinafter, operation examples 1 to 5 of the sound signal processing method performed by the sound signal processing device 100 will be described.

[0238] [Action Example 1]

[0239] Figure 4 This is a flowchart of operation example 1 of the sound signal processing device 100 according to the present embodiment.

[0240] First, the acquisition unit 110 acquires first position information indicating the position of an object as a moving body in the virtual space, second position information indicating the position of a listener L in the virtual space, and ability information indicating whether the object generates aerodynamic sound (S10). Note that step S10 corresponds to an acquisition step.

[0241] In addition, in Operation Example 1, shape information is not used, and therefore in step S10 , the acquisition unit 110 does not acquire shape information.

[0242] Furthermore, the acquisition unit 110 continuously acquires the first position information and the second position information, more specifically, each time the spatial information is updated by the spatial information management units A0201 and A0211. That is, the acquisition unit 110 repeatedly acquires the first position information and the second position information each time the spatial information is updated by the spatial information management units A0201 and A0211. Each time the acquisition unit 110 acquires the first position information and the second position information, the acquisition unit 110 outputs the first position information and the second position information to the calculation unit 120, or after acquiring a certain number of the first position information and the second position information, the acquisition unit 110 outputs the certain number of the first position information and the certain number of the second position information to the calculation unit 120. That is, the acquisition unit 110 outputs a plurality of first position information and a plurality of second position information to the calculation unit 120. Furthermore, the acquisition unit 110 outputs the acquired availability information to the generation unit 130 or the selection unit (not shown).

[0243] Next, the calculation unit 120 (more specifically, the first calculation unit 121) calculates the moving speed of the object in the virtual space based on the first position information acquired by the acquisition unit 110 (S20). In addition, this step S20 corresponds to the first calculation step. The first calculation unit 121 outputs the calculated moving speed to the generation unit 130.

[0244] Next, the calculation unit 120 (more specifically, the second calculation unit 122) calculates the distance between the object and the listener L based on the first position information obtained by the acquisition unit 110 and the second position information obtained (S30). Note that this step S30 corresponds to the second calculation step. The second calculation unit 122 outputs the calculated moving speed to the generation unit 130.

[0245] Next, the generator 130 determines whether the availability information acquired by the acquirer 110 indicates that the target object can generate pneumatic sound ( S40 ).

[0246] If the object does not generate aerodynamic sound (No in step S40), the acoustic signal processing device 100 ends the operation. If the object generates aerodynamic sound (Yes in step S40), the generator 130 generates an aerodynamic sound signal (S50). More specifically, the generator 130 generates an aerodynamic sound signal based on the moving speed calculated by the first calculator 121 and the distance calculated by the second calculator 122. The aerodynamic sound signal indicates the aerodynamic sound generated by the wind W generated by the movement of the object and reaches the ear of the listener L. In addition, step S50 corresponds to the generation step.

[0247] Next, the output unit 140 outputs the pneumatic sound signal generated by the generator 130 (S60). Step S60 corresponds to an output step. Here, the output unit 140 outputs the pneumatic sound signal to the earphone 200. Thus, the earphone 200 can reproduce the pneumatic sound represented by the output pneumatic sound signal.

[0248] Here, in step S50 , the following first and second examples will be described as a method for the generation unit 130 to generate the pneumatic sound signal.

[0249] <Example 1>

[0250] First, the first example will be described. In the first example, the generator 130 generates the pneumatic sound signal based on the pneumatic sound data in addition to the moving speed and the distance.

[0251] Figure 5 and Figure 6 It means in Figure 4 The flowchart of the first example of the method in which the generator 130 generates the pneumatic sound signal in step S50 is shown.

[0252] like Figure 5 As shown in FIG. 1 , when the generator 130 generates the pneumatic sound signal by the method of the first example, the pneumatic sound data is stored in the storage unit 150 before step S10 (S70). Then, after steps S10 to S40 are performed as described above, Figure 6 As shown, the generator 130 generates a pneumatic sound signal based on the pneumatic sound data stored in the storage unit 150 , the moving speed calculated by the first calculator 121 , and the distance calculated by the second calculator 122 ( S51 ).

[0253] More specifically, in step S51 , the generator 130 first acquires the pneumatic sound data stored in the storage unit 150 ( S511 ).

[0254] The pneumatic sound data is data obtained by collecting the sound generated by the wind W reaching the human ear or a model simulating the human ear. In this operation example, the pneumatic sound data is data obtained by collecting the sound generated by the wind W reaching the model simulating the human ear.

[0255] Here, use Figure 7 and Figure 8 The aerodynamic sound data will be described. Figure 7 1 is a schematic diagram of a pseudo-human head microphone 10 for collecting aerodynamic sound data according to the method of the first example of operation example 1. Figure 8 1 is a schematic diagram of a pseudo-human head microphone 20 for collecting aerodynamic sound data according to the method of the first example of operation example 1.

[0256] Figure 7 The illustrated human-like head microphone 10 includes a human-like head 11 and microphones 12 and 13 respectively disposed at the two ears of the human-like head 11. Figure 8 The illustrated anthropomorphic microphone 20 is a simple anthropomorphic microphone, and a microphone is provided on a model simulating a human ear. By radiating wind W from, for example, an electric fan 50 to such anthropomorphic microphones 10 and 20, the anthropomorphic microphones 10 and 20 can collect aerodynamic sound generated by the wind W reaching the ear of the listener L. The aerodynamic sound data is data (sound signal) representing the collected aerodynamic sound.

[0257] Next, the generator 130 generates a pneumatic sound signal in such a manner that the frequency component is shifted toward the high frequency side as the calculated moving speed is larger, and the volume of the pneumatic sound is increased as the calculated distance is smaller (S512). That is, the generated pneumatic sound signal is a signal in which the frequency component is shifted toward the high frequency side as the calculated moving speed is larger, and the volume of the pneumatic sound is increased as the calculated distance is smaller.

[0258] More specifically, the generator 130 generates a pneumatic sound signal in which the frequency component is shifted toward a higher frequency side as the moving speed calculated by the first calculator 121 is larger based on the pneumatic sound data. Here, the generator 130 generates a pneumatic sound signal in which the frequency component included in the pneumatic sound data is shifted toward a higher frequency side as the moving speed is larger. For example, the generator 130 shifts the frequency toward a higher frequency side by multiplying the frequency of a certain frequency component by a value corresponding to the moving speed.

[0259] Furthermore, the generator 130 generates the pneumatic sound signal based on the pneumatic sound data in such a manner that the smaller the distance calculated by the second calculator 122 is, the larger the volume of the pneumatic sound is. For example, a predetermined distance is stored in the storage unit 150, and the generator 130 generates the pneumatic sound signal in such a manner that the smaller the distance calculated compared to the predetermined distance is, the larger the volume of the pneumatic sound is, and in such a manner that the larger the distance calculated compared to the predetermined distance is, the smaller the volume of the pneumatic sound is. Furthermore, for example, the generator 130 generates the pneumatic sound signal in such a manner that the volume of the pneumatic sound is attenuated by a (a is a constant)th power of the distance.

[0260] In the first example, the generator 130 performs both the above-mentioned processing of shifting the frequency component to the high frequency side and the above-mentioned processing of increasing the volume of the aerodynamic sound, but may perform at least one of them.

[0261] In the first example, the aerodynamic sound data is data obtained by collecting the sound generated by the wind W reaching the model simulating the human ear, but it may also be data obtained by collecting the sound generated by the wind W reaching the human ear. In this case, microphones may be provided at both ears of the human, for example, by radiating the wind W from the electric fan 50, and the microphones collect the aerodynamic sound generated by the wind W reaching the ears of the listener L. The aerodynamic sound data is data (sound signal) representing the collected aerodynamic sound.

[0262] <Example 2>

[0263] Next, a second example of a method in which the generator 130 generates the pneumatic sound signal will be described.

[0264] Fig. 9 It means in Figure 4 The flowchart of the second example of the method in which the generator 130 generates the pneumatic sound signal in step S50 is shown.

[0265] After step S40 is performed as described above, Fig. 9 As shown, the generator 130 generates an aerodynamic sound signal (S52). Here, the generator 130 generates an aerodynamic sound signal based on the moving speed calculated by the first calculator 121, the distance calculated by the second calculator 122, and the shape of the human ear.

[0266] More specifically, in step S52 , first, the generator 130 generates a processing signal based on the moving speed calculated by the first calculator 121 and the shape of the human ear ( S521 ).

[0267] In step S521, the following processing is performed. The generator 130 processes the noise signal using a plurality of frequency band emphasis filters, and multiplies the noise signal processed by the frequency band emphasis filter among the processed plurality of noise signals by a predetermined coefficient corresponding to the frequency band emphasis filter (S521a). Next, the generator 130 synthesizes the plurality of noise signals multiplied by the predetermined coefficient to generate a processed signal (S521b).

[0268] Fig. 10A This is a diagram for explaining a process of generating a pneumatic sound signal by the generator 130 according to the method of the second example of Operation Example 1.

[0269] As described above, wind W is generated by the movement of objects in the virtual space. Fig. 10A As shown, aerodynamic sound is generated when the generated wind W reaches the human ear. More specifically, when the generated wind W reaches the human ear, an air vortex (Karman vortex) is generated, and aerodynamic sound is generated by the Karman vortex. Here, the human ear is regarded as a collection of depressions with multiple widths (for example, the first width d1, the second width d2, the third width d3 and the fourth width d4), and the generation unit 130 generates an aerodynamic sound signal by applying a cavity sound generation model. In addition, the Karman vortex frequency at the part of the human ear corresponding to the first width d1, for example, is represented by f1, which satisfies equation (1).

[0270] f1=(St×v)÷d1 Formula (1)

[0271] Similarly, the Karman vortex frequencies at the parts of the human ear corresponding to the second width d2, the third width d3, and the fourth width d4 are represented by f2, f3, and f4, respectively, satisfying equations (2), (3), and (4).

[0272] f2=(St×v)÷d2 Formula (2)

[0273] f3=(St×v)÷d3 Formula (3)

[0274] f4=(St×v)÷d4 Formula (4)

[0275] For example, the first to fourth widths d1 to d4 may be the size of an average human ear, for example, a value of 0 cm or more and 3 cm or less. Fig. 10AThe human ear shown is also modeled on an average human ear. The moving speed of the object calculated by the first calculation unit 121 can be regarded as the wind speed v (m / s) of the wind W generated by the movement of the object.

[0276] Furthermore, St is the Struch number, which is about 0.4 as an example in the cavity sound generation model. In addition, the Karman vortex frequency is equivalent to the frequency of the aerodynamic sound. The generator 130 calculates f1 to f4 as the Karman vortex frequencies according to equations (1) to (4). The generator 130 generates a processed signal by processing the noise signal using four bandpass filters with f1 to f4 as their respective center frequencies.

[0277] also, Fig. 10A The first to fourth widths d1 to d4 shown in FIG. 1 represent the widths when the wind W comes from the front of the listener L. Here, Fig. 10B A case where the wind W comes from a direction other than the front of the listener L will be described.

[0278] Fig. 10B 1 is another diagram for explaining the process of generating the pneumatic sound signal by the generating unit of the method of the second example of the operation example 1. More specifically, Fig. 10B The figure shows the situation where the wind W comes from 45 degrees above the listener L. In this case, Fig. 10B In this way, the width changes in appearance according to the angle, so the frequency characteristics obviously change accordingly.

[0279] Fig.11 This is a diagram showing an example of four bandpass filters and their processing in the method of the second example of operation example 1. The four bandpass filters are filters that pass frequency components with f1 to f4 as center frequencies, that is, they are an example of frequency band emphasis filters for simulating aerodynamic sound generated by the shape of the ear of the listener L.

[0280] The generating unit 130 converts the noise signal into Fig.11 The frequency components of the four band-pass filters shown are extracted and synthesized as processing signals. In addition, the center frequency is also the frequency of the peak value of the gain of the band-pass filter. In addition, in the second example, as the noise signals respectively input to the four band-pass filters, white noise signals or pink noise signals can be used.

[0281] In addition, the output signals output from the four bandpass filters may be multiplied by a predetermined coefficient corresponding to the center frequency. In addition, the output signal refers to the processed noise signal. The predetermined multiplier may also be a value that is larger as the center frequency is lower. This is because, in a typical frequency band of aerodynamic sound (e.g., below about 5 kHz), the lower the frequency, the smaller the loudness perceived by the listener L. The predetermined multiplier is a value set corresponding to each of the multiple bandpass filters (multiple frequency band emphasis filters), and is multiplied by the noise signal processed by the bandpass filter.

[0282] In addition, the characteristics of each of the four bandpass filters (multiple band emphasis filters) may vary according to the wind speed of the wind W reaching the ears of the listener L, and the characteristics may also vary according to the wind direction of the wind W reaching the ears of the listener L.

[0283] Furthermore, if Fig. 9 As shown, the generator 130 generates the pneumatic sound signal such that the frequency component is shifted toward the high frequency side as the calculated moving speed is larger, and the volume of the pneumatic sound is increased as the calculated distance is smaller (S512).

[0284] In the second example, the generator 130 also generates an aerodynamic sound signal in which the frequency of the frequency component included in the synthesized processed signal is shifted toward the high frequency side as the moving speed is greater. Furthermore, the generator 130 generates the aerodynamic sound signal in such a manner that the volume of the aerodynamic sound is increased as the distance calculated by the second calculator 122 is smaller. That is, in addition to the aerodynamic sound data shown in the first example, the processed signal is also used for processing in the second example. In the second example, the generator 130 performs both the above-mentioned processing of shifting the frequency component toward the high frequency side and the above-mentioned processing of increasing the volume of the aerodynamic sound, but at least one of them may be performed.

[0285] In the above description, an example of applying a cavity sound generation model to a human ear is described, but the present invention is not limited to this. Fig.12 and Fig.13 This is another diagram for explaining the process of generating a processing signal by the generation unit 130 according to the method of the second example of the operation example 1.

[0286] The generating unit 130 may also be as follows Fig.12 As shown, the processing signal is generated by assuming that the human ear is a cylindrical shape. In this case, the frequency of the Karman vortex generated by the ear of size d5 (Karman vortex frequency) is represented by f5, which satisfies equation (5).

[0287] f5=(St×v)÷d5 Formula (5)

[0288] In this model, St, which is the Strouhal number, is approximately 0.2 as an example. Then, the generator 130 generates a processed signal by processing the noise signal using a bandpass filter having a center frequency of f5, as described above.

[0289] In addition, the generating unit 130 may also Fig.13 As shown, the processing signal is generated by assuming that the human head is a cylindrical shape. In this case, the frequency of the Karman vortex generated by the head of size d6 (Karman vortex frequency) is represented by f6, which satisfies equation (6).

[0290] f6=(St×v)÷d6 Formula (6)

[0291] In this model, St, which is the Strouhal number, is about 0.2 as an example. In addition, the generator 130 generates a processed signal by processing the noise signal with a bandpass filter having a center frequency of f6 as described above. In addition, the bandpass filter is a filter for simulating aerodynamic sound generated by the shape of the head of the listener L.

[0292] In this way, the generator 130 generates the pneumatic sound signal by the methods of the first example and the second example, and outputs the pneumatic sound signal to the output unit 140 .

[0293] Next, the process of step S60 is performed.

[0294] [Action Example 2]

[0295] Next, operation example 2 will be described.

[0296] Fig.14 This is a flowchart of Operation Example 2 of the sound signal processing device 100 according to the present embodiment. The following description will focus on differences from Operation Example 1, and description of common points will be omitted or simplified.

[0297] First, the acquisition unit 110 acquires first position information indicating the position of an object as a moving body in the virtual space, second position information indicating the position of a listener L in the virtual space, and shape information indicating the shape of the object ( S10 a ).

[0298] In addition, in the operation example 2, the availability information is not used, and therefore in step S10 a , the acquisition unit 110 does not acquire the availability information.

[0299] In addition, similarly to Operation Example 1, the acquisition unit 110 acquires the first position information and the second position information continuously, more specifically, every time the spatial information management units A0201 and A0211 update the spatial information.

[0300] Next, steps S20 and S30 are performed.

[0301] Next, the generator 130 generates an aerodynamic sound signal based on the shape indicated by the shape information acquired by the acquirer 110 , the moving speed calculated by the first calculator 121 , and the distance calculated by the second calculator 122 ( 50 a ).

[0302] In step S50a, the generator 130 first acquires the pneumatic sound data stored in the storage unit 150 (S511).

[0303] Next, the generator 130 generates a pneumatic sound signal in such a manner that the frequency component is shifted toward the high frequency side as the moving speed is greater, the volume of the pneumatic sound is increased as the distance is smaller, and the volume of the pneumatic sound is increased as the volume of the object is larger (S512a). That is, the generated pneumatic sound signal is a signal in which the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, a signal in which the volume of the pneumatic sound is increased as the calculated distance is smaller, and a signal in which the volume of the pneumatic sound is increased as the volume of the object is larger. In addition, the volume of the object is calculated based on the shape represented by the shape information.

[0304] In the second operation example, the generator 130 performs all of the above-mentioned processing of shifting the frequency component to the high frequency side, the above-mentioned processing of increasing the volume of the pneumatic sound according to the distance, and the above-mentioned processing of increasing the volume of the pneumatic sound according to the volume of the object, but at least one of them may be performed. In addition, the above-mentioned processing of increasing the volume of the pneumatic sound according to the volume of the object may be performed as follows. For example, a predetermined volume is stored in the storage unit 150, and the generator 130 generates the pneumatic sound signal in such a manner that the larger the volume of the object is compared with the predetermined volume, the larger the volume of the pneumatic sound is, and the smaller the volume of the pneumatic sound is compared with the predetermined volume, the smaller the volume of the pneumatic sound is.

[0305] Furthermore, the output unit 140 outputs the pneumatic sound signal generated by the generating unit 130 ( S60 ).

[0306] In addition, Fig.14 In the illustrated operation example 2, the pneumatic sound signal is generated using the pneumatic sound data of the first example method illustrated in operation example 1, but the present invention is not limited thereto. In operation example 2, the pneumatic sound signal may be generated using the processed signal of the second example method illustrated in operation example 1.

[0307] [Action Example 3]

[0308] Next, operation example 3 will be described.

[0309] Fig.15 This is a flowchart of Operation Example 3 of the sound signal processing device 100 according to the present embodiment. The following description will focus on differences from Operation Example 2, and description of common points will be omitted or simplified.

[0310] In the operation example 3, steps S10a, S20, and S30 are performed in the same manner as in the operation example 2.

[0311] Next, the calculation unit 120 (more specifically, the third calculation unit 123 ) calculates the air volume per unit time generated when the object moves, based on the moving speed calculated by the first calculation unit 121 and the shape information acquired by the acquisition unit 110 ( S35 ).

[0312] Here, the wind volume refers to the amount of air moved due to the movement of the object. For example, the moving speed of the object, the shape of the object, and wind volume data indicating the wind volume corresponding to the moving speed and the shape are stored in the storage unit 150. The third calculation unit 123 can calculate the wind volume based on the moving speed and the shape by referring to the wind volume data stored in the storage unit 150.

[0313] Next, the generator 130 generates an aerodynamic sound signal ( 50 b ) based on the moving speed calculated by the first calculator 121 , the distance calculated by the second calculator 122 , and the air volume calculated by the third calculator 123 .

[0314] In step S50b, the generator 130 first acquires the pneumatic sound data stored in the storage unit 150 (S511).

[0315] Next, the generator 130 generates a pneumatic sound signal in such a manner that the frequency component is shifted toward the high frequency side as the moving speed is greater, the volume of the pneumatic sound is increased as the distance is smaller, and the volume of the pneumatic sound is increased as the air volume is greater (S512b). That is, the generated pneumatic sound signal is a signal in which the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, a signal in which the volume of the pneumatic sound is increased as the calculated distance is smaller, and a signal in which the volume of the pneumatic sound is increased as the air volume is greater.

[0316] In the third operation example, the generator 130 performs all of the above-mentioned processing of shifting the frequency component to the high frequency side, the above-mentioned processing of increasing the volume of the aerodynamic sound according to the distance, and the above-mentioned processing of increasing the volume of the aerodynamic sound according to the air volume, but at least one of them may be performed. In addition, the above-mentioned processing of increasing the volume of the aerodynamic sound according to the air volume may be performed as follows. For example, the specified air volume is stored in the storage unit 150, and the generator 130 generates the aerodynamic sound signal in such a manner that the larger the air volume calculated compared to the specified air volume is, the larger the volume of the aerodynamic sound is, and in such a manner that the smaller the air volume calculated compared to the specified air volume is, the smaller the volume of the aerodynamic sound is.

[0317] Furthermore, the output unit 140 outputs the pneumatic sound signal generated by the generating unit 130 ( S60 ).

[0318] [Action Example 4]

[0319] Next, operation example 4 will be described.

[0320] In the operation example 3, in step S35, the air volume per unit time generated when the object moves is calculated using the air volume data, but the present invention is not limited to this.

[0321] In operation example 4, an example of performing processing according to the moving direction (advancing direction) of an object in a virtual space will be described.

[0322] Fig.16 This is a flowchart of Operation Example 4 of the sound signal processing device 100 according to the present embodiment. The following description will focus on differences from Operation Example 3, and description of common points will be omitted or simplified.

[0323] In the operation example 4, similarly to the operation example 3, steps S10a, S20, and S30 are performed.

[0324] Next, the process proceeds to step S35. The process performed in step S35 will be described in more detail below.

[0325] In this operation example, the object is the aircraft 700 . Fig.17 FIG. 1 is a diagram showing an aircraft 700 as an object in operation example 4 of the sound signal processing device according to the present embodiment. In this way, when the object is the aircraft 700, the shape information indicates the moving direction of the object. Fig.17 , the moving direction (traveling direction) of the object (aircraft 700) is the front direction of the paper.

[0326] In this case, in step S35, the following processing is performed as a more detailed processing: That is, the third calculation unit 123 calculates the air volume based on the area of ​​the object viewed from the moving direction of the object indicated by the shape information acquired by the acquisition unit 110 and the moving speed calculated by the first calculation unit 121 (S35c).

[0327] This area is an area surrounded by the outline of the object when the object is viewed from the moving direction (traveling direction) of the aircraft 700 as the object. This area is, for example, Fig.17 The range occupied by aircraft 700 is shown.

[0328] In step S35c, the third calculation unit 123 calculates the air volume by multiplying the area of ​​the object by the moving speed. More specifically, the third calculation unit 123 may calculate the air volume by multiplying the area of ​​the object, the moving speed, and a predetermined constant.

[0329] Then, if Fig.16 As shown, steps S50b and S60 are performed.

[0330] In this operation example, the aircraft 700 is used as the object, but the present invention is not limited thereto. The processing of this operation example can be applied when using an object whose moving direction (travel direction) is obvious or an object whose moving direction (travel direction) is indicated by shape information.

[0331] [Action Example 5]

[0332] Next, operation example 5 will be described.

[0333] First, the objects in Action Example 5 are described.

[0334] Fig.18 FIG. 8 is a diagram showing a fan 800 as a target in operation example 5 of the sound signal processing device according to the present embodiment. Unlike the airplane 700 in operation example 4, the fan 800 has an unclear moving direction. Fig.18 At least one of the up-and-down direction on the paper, the left-and-right direction on the paper, and the approach-to-depth direction on the paper.

[0335] Thus, when using an object whose moving direction is not obvious, it is difficult to perform the processing of Operation Example 4. In Operation Example 5, an example of calculating the air volume per unit time generated when the object moves is described in such a case as well.

[0336] Fig.19 This is a flowchart of Operation Example 5 of the sound signal processing device 100 according to the present embodiment. The following description will focus on differences from Operation Example 3, and description of common points will be omitted or simplified.

[0337] In the operation example 5, step S10a is performed in the same manner as in the operation example 3.

[0338] Next, the first calculation unit 121 calculates the moving speed and moving direction of the object based on the first position information acquired by the acquisition unit 110 (S20e). As described above, the acquisition unit 110 acquires the first position information indicating the position of the object in the virtual space each time the spatial information is updated by the spatial information management units A0201 and A0211. The first calculation unit 121 calculates the moving direction of the object in the virtual space based on the difference between the first position information and the first position information acquired after the spatial information is updated by the spatial information management units A0201 and A0211.

[0339] Next, proceed to step S30.

[0340] Next, the third calculation unit 123 calculates the air volume per unit time generated when the object moves, based on the shape information acquired by the acquisition unit 110 and the moving direction and moving speed calculated by the first calculation unit 121 ( S35e ).

[0341] In step S35e, first, the third calculation unit 123 calculates the area of ​​the object observed from the moving direction of the object based on the shape information and the moving direction. In addition, the round fan 800 as the object has a handle 801 and a fan surface 802. The situation where the round fan 800 moves in the up and down direction (hereinafter referred to as the first direction), the left and right direction (hereinafter referred to as the second direction), and the paper surface approach depth direction (hereinafter referred to as the third direction) is studied. In addition, the first direction is a direction horizontal to the fan surface 802, and is a direction in which the handle 801 extends. The second direction is a direction horizontal to the fan surface 802, and is a direction perpendicular to the first direction. The third direction is a direction perpendicular to the fan surface 802. As Fig.18 As shown, the round fan 800 is a round fan having a relatively thin thickness in the third direction.

[0342] In the following, when the moving direction of the round fan 800 is the first direction, the area of ​​the object observed from the moving direction is referred to as the first area. In the following, when the moving direction of the round fan 800 is the second direction, the area of ​​the object observed from the moving direction is referred to as the second area. In the following, when the moving direction of the round fan 800 is the third direction, the area of ​​the object observed from the moving direction is referred to as the third area. Since the thickness of the round fan 800 is thin in the third direction, the first area and the second area are very small values ​​compared to the third area.

[0343] Furthermore, in step S35e, the third calculation unit 123 calculates the air volume based on the area and moving speed of the object observed from the moving direction of the object. For example, similar to step S35 of action example 4, the third calculation unit 123 can calculate the air volume by multiplying the area and moving speed of the object by a predetermined constant.

[0344] For example, in this action example, since the first area and the second area are very small values ​​compared to the third area, the wind volume is larger when the moving direction of the fan 800 is the third direction than when the moving direction of the fan 800 is the first direction and the second direction.

[0345] Then, if Fig.19 As shown, steps S50b and S60 are performed.

[0346] Next, pipeline processing will be described.

[0347] The processing performed by the calculation unit 120 , the generation unit 130 , and the selection unit described above may be performed as a part of pipeline processing as described in Patent Document 2, for example. Fig. 20 It is used to illustrate Figure 2G and Figure 2H This is a diagram showing an example of a functional block diagram and steps for the case where the rendering units A0203 and A0213 perform pipeline processing. Fig. 20 In the description, use Figure 2G and Figure 2H The rendering unit 900 is described as an example of the rendering units A0203 and A0213.

[0348] Pipeline processing means dividing the process for providing an acoustic effect into a plurality of processes and executing each process in sequence. In each of the divided processes, for example, signal processing of a sound signal or generation of parameters used in the signal processing is performed.

[0349] The rendering unit 900 of the present embodiment includes implementing processes such as reverberation effects, initial reflected sound, distance attenuation effects, binaural processing, etc. as pipeline processing. However, the above-mentioned processing is an example, and other processing may be included, or some processing may not be included. For example, the rendering unit 900 may include diffraction processing or occlusion processing as pipeline processing, and it may be omitted when reverberation processing is not required. In addition, each processing may be represented as a stage, and the sound signal such as reflected sound generated as a result of each processing may be represented as a rendering item. The order of each stage in the pipeline processing and the stages included in the pipeline processing are not limited to Fig. 20 Example shown.

[0350] In addition, the rendering unit 900 may not include Fig. 20 All the stages shown in , some of the stages may be omitted or other stages may exist in addition to the rendering unit 900.

[0351] As an example of pipeline processing, the processing performed in each of reverberation processing, initial reflection processing, distance attenuation processing, selection processing, generation processing, and binaural processing is described. In each process, metadata included in the input signal is analyzed and parameters required for generating reflected sound are calculated.

[0352] In addition, Fig. 20 In the embodiment, the rendering unit 900 includes a reverberation processing unit 901, an initial reflection processing unit 902, a distance attenuation processing unit 903, a selection unit 904, the calculation unit 120, the generation unit 130, and a binaural processing unit 905. Here, an example is described in which the reverberation processing unit 901 performs a reverberation processing step, the initial reflection processing unit 902 performs an initial reflection processing step, the distance attenuation processing unit 903 performs a distance attenuation processing step, the selection unit 904 performs a selection processing step, and the binaural processing unit 905 performs a binaural processing step.

[0353] In the reverberation processing step, the reverberation processing unit 901 generates a sound signal representing a reverberation sound or a parameter required for generating a sound signal. The reverberation sound is a sound that is included in the reverberation sound that reaches the listener as a reverberation after the direct sound. As an example, the reverberation sound is a reverberation sound that reaches the listener after a relatively late period (for example, about a hundred and several dozen ms from the arrival of the direct sound) after the initial reflected sound described later reaches the listener, after being reflected more times (for example, several dozen times) than the initial reflected sound. The reverberation processing unit 901 refers to the sound signal and spatial information included in the input signal, and uses a predetermined function prepared in advance for generating the reverberation sound to calculate.

[0354] The reverberation processing unit 901 may also apply a known reverberation generation method to the sound signal to generate reverberation. As an example, the known reverberation generation method is the Schroeder method, but is not limited thereto. In addition, when applying the known reverberation generation process, the reverberation processing unit 901 uses the shape and acoustic characteristics of the sound reproduction space represented by the spatial information. Thus, the reverberation processing unit 901 can calculate parameters for generating a sound signal representing the reverberation.

[0355] In the initial reflection processing step, the initial reflection processing unit 902 calculates the parameters used to generate the initial reflected sound based on the spatial information. The initial reflected sound is the reflected sound that reaches the listener after one or more reflections in the relatively early period (for example, about tens of ms from the arrival of the direct sound) after the direct sound reaches the listener from the sound source object. The initial reflection processing unit 902, for example, refers to the sound signal and metadata, and uses the shape, size, position of objects such as structures and the reflectivity of the objects in the three-dimensional sound field (space) to calculate the path of the reflected sound that is reflected from the sound source object and reaches the listener. In addition, the initial reflection processing unit 902 can also calculate the path of the direct sound. The information of the path can also be used as a parameter for generating the initial reflected sound, and can also be used as a parameter for the selection processing of the reflected sound in the selection unit 904.

[0356] In the distance attenuation processing step, the distance attenuation processing unit 903 calculates the volume reaching the listener based on the difference between the length of the direct sound path and the length of the reflected sound path calculated by the initial reflection processing unit 902. The volume reaching the listener is attenuated in proportion to the distance to the listener (inversely proportional to the distance) relative to the volume of the sound source, so the volume of the direct sound can be obtained by dividing the volume of the sound source by the length of the direct sound path, and the volume of the reflected sound can be calculated by dividing the volume of the sound source by the length of the reflected sound path.

[0357] In the selection process step, the selection unit 904 selects aerodynamic sound or reflected sound to be generated. The selection process may be performed based on the parameters calculated in the previous step.

[0358] When the selection process is performed as part of the pipeline process, the reflected sound that is not selected in the selection process may not be the subject of the process after the selection process in the pipeline process. By not performing the process after the selection process on the reflected sound that is not selected, the computational load of the sound signal processing device 100 can be reduced compared to the case where it is decided not to perform the binaural process on the reflected sound that is not selected.

[0359] Furthermore, when the selection process described in this embodiment is executed by a part of the pipeline process, if the order of the selection process is set to be executed in an earlier order among the orders of the plurality of processes in the pipeline process, more processes after the selection process can be omitted, so that the amount of calculation can be reduced. For example, if the calculation unit 120 and the generation unit 130 execute the selection process in an earlier order than the process, the process related to the aerodynamic sound related to the object determined not to be selected can be omitted, and the amount of calculation in the acoustic signal processing device 100 can be further reduced.

[0360] Furthermore, the parameters calculated in a part of the pipeline process for generating the rendering item may be used in the selection unit 904 or the calculation unit 120 .

[0361] In the binaural processing step, the binaural processing unit 905 performs signal processing on the sound signal of the direct sound so that it is perceived as sound reaching the listener from the direction of the sound source object. Furthermore, the binaural processing unit 905 performs signal processing so that the reflected sound is perceived as sound reaching the listener from an obstacle object related to the reflection. Based on the coordinates and orientation of the listener in the sound space (i.e., the position and orientation of the listening point), processing of applying HRIR (Head-Related Impulse Responses) DB (Data base) is performed so that the sound reaches the listener from the position of the sound source object or the position of the obstacle object. In addition, the position and direction of the listening point can change, for example, to match the movement of the listener's head. In addition, information indicating the position of the listener can also be obtained from the sensor.

[0362] The programs used in pipeline processing and binaural processing, spatial information required for sound processing, HRIR DB, threshold data and other parameters are obtained from the memory of the sound signal processing device 100 or from the outside. HRIR (Head-Related Impulse Responses) is the response characteristic when one impulse is generated. In other words, HRIR is the response characteristic obtained by Fourier transforming the head-related transfer function that expresses the change of sound generated by the surrounding objects including the ear shell, the human head and shoulders as a transfer function, thereby transforming the expression from the frequency domain to the time domain. HRIR DB is a database containing such information.

[0363] In addition, as an example of pipeline processing, the rendering unit 900 may include a processing unit not shown in the figure, for example, a diffraction processing unit or an occlusion processing unit.

[0364] The diffraction processing unit performs processing to generate a sound signal representing a sound including diffracted sound caused by an obstacle between a listener and a sound source object in a three-dimensional sound field (space). The diffracted sound is a sound that, when there is an obstacle between the sound source object and the listener, bypasses the obstacle and reaches the listener from the sound source object.

[0365] The diffraction processing unit, for example, refers to the sound signal and metadata, and uses the position of the sound source object in the three-dimensional sound field (space), the position of the listener, and the position, shape and size of the obstacle to calculate the path from the sound source object to bypass the obstacle and reach the listener, and generates diffracted sound based on the path.

[0366] The occlusion processing unit generates a sound signal that can be faintly heard when there is a sound source object on the opposite side of the obstacle object, based on the space information acquired in a certain step and information such as the material of the obstacle object.

[0367] In addition, in the above-mentioned embodiment, the position information assigned to the sound source object is defined as the information of a "point" in the virtual space, and the details of the invention are described assuming a so-called "point sound source". On the other hand, as a method of defining a sound source in a virtual space, there is also a case where a sound source that is not a point sound source and is spatially extended is defined as an object having length, size or shape. In such a case, since the distance between the listener and the sound source or the direction of arrival of the sound is uncertain, the reflected sound generated thereby does not need to be analyzed, or regardless of the analysis result, it can be limited to the processing of "selection" in the above-mentioned selection unit 904. This is because, by doing so, it is possible to avoid the degradation of sound quality that may be caused by not selecting the reflected sound. Alternatively, a representative point such as the center of gravity of the object can be set, and the processing of the present disclosure can be applied assuming that the sound is generated from the representative point. In this case, the processing of the present disclosure can also be applied after adjusting the threshold according to the information of the spatial extension of the sound source.

[0368] Next, an example of the construction of a bit stream is described.

[0369] The bitstream includes, for example, a sound signal and metadata. The sound signal is sound data that represents the sound, indicating information related to the frequency and strength of the sound. The spatial information included in the metadata is information related to the space in which the listener who hears the sound based on the sound signal is located. Specifically, the spatial information is information related to the predetermined position (localization position) when the sound image of the sound is localized at a predetermined position in the sound space (for example, in a three-dimensional sound field), that is, when the listener perceives the sound as arriving from a predetermined direction. The spatial information includes, for example, sound source object information and position information indicating the position of the listener.

[0370] The sound source object information is information about an object that generates sound based on a sound signal, that is, an object that reproduces the sound signal, and is information about a virtual object (sound source object) arranged in a virtual space corresponding to the real space in which the object is arranged, that is, a sound space. The sound source object information includes, for example, information indicating the position of the sound source object arranged in the sound space, information about the direction of the sound source object, information about the directionality of the sound emitted by the sound source object, information indicating whether the sound source object is a living thing, and information indicating whether the sound source object is a moving body. For example, a sound signal corresponds to one or more sound source objects indicated by the sound source object information.

[0371] As an example of the data structure of a bit stream, the bit stream is composed of, for example, metadata (control information) and an audio signal.

[0372] The sound signal and metadata can be stored in one bitstream or in multiple bitstreams. Similarly, the sound signal and metadata can be stored in one file or in multiple files.

[0373] The bit stream may exist for each sound source or for each playback time. When a bit stream exists for each playback time, a plurality of bit streams may be processed in parallel at the same time.

[0374] The metadata may be assigned to each bit stream or may be assigned as information for controlling a plurality of bit streams. Furthermore, the metadata may be assigned to each playback time.

[0375] When the sound signal and metadata are stored in a plurality of bitstreams or files, they may include information indicating other bitstreams or files associated with one or a part of the bitstreams or files, or may include information indicating other bitstreams or files associated with all the bitstreams or files. Here, the associated bitstream or file may be, for example, a bitstream or file that may be used simultaneously during audio processing. In addition, the associated bitstream or file may include a bitstream or file in which information indicating other associated bitstreams or files is recorded together. Here, the information indicating other associated bitstreams or files may be, for example, an identifier indicating the other bitstream, a file name indicating other files, a URL (Uniform Resource Locator) or a URI (Uniform Resource Identifier), etc. In this case, the acquisition unit 110 determines or acquires the bitstream or file based on the information indicating other associated bitstreams or files. In addition, the bitstream may include information indicating other associated bitstreams, and the bitstream may include information indicating bitstreams or files associated with other bitstreams or files. Here, the file including information indicating the associated bitstream or file may be, for example, a control file such as a manifest file used for content distribution.

[0376] In addition, all or part of the metadata may be obtained from outside the bitstream of the audio signal. For example, either metadata for controlling the audio or metadata for controlling the video may be obtained from outside the bitstream, or both metadata may be obtained from outside the bitstream. Furthermore, when metadata for controlling the video is included in the bitstream obtained by the audio signal reproduction system, the audio signal reproduction system may also have a function of outputting metadata that can be used for controlling the video to a display device that displays an image or a stereoscopic video reproduction device that reproduces a stereoscopic video.

[0377] Furthermore, examples of information included in metadata will be described.

[0378] Metadata may also be information used to describe a scene represented by a sound space. Here, a scene is a term used to represent a collection of all elements of a three-dimensional image and sound event in a sound space modeled by a sound signal reproduction system using metadata. That is, the metadata described here includes not only information for controlling sound processing, but also information for controlling image processing. Of course, metadata may include information for controlling only one of the sound processing and the image processing, or information used for controlling both.

[0379] The sound signal reproduction system performs sound processing on the sound signal by using metadata included in the bitstream and additional interactive listener position information, etc., to generate a virtual sound effect. In this embodiment, the sound effects are described as performing initial reflection processing, obstacle processing, diffraction processing, occlusion processing, and reverberation processing, but other sound processing can also be performed using metadata. For example, it can be considered that the sound signal reproduction system adds sound effects such as distance attenuation effect, positioning, and Doppler effect. In addition, information for switching on and off all or part of the sound effects and priority information can also be added as metadata.

[0380] In addition, as an example, the encoded metadata includes: information related to the sound space containing the sound source object and the obstacle object, and information related to the positioning position when the sound image of the sound is positioned at a specified position in the sound space (that is, so that it is perceived as a sound arriving from a specified direction). Here, the obstacle object is an object that may affect the sound perceived by the listener, such as blocking or reflecting the sound during the period from the sound emitted by the sound source object to the sound reaching the listener. In addition to stationary objects, obstacle objects may also include animals such as humans or moving objects such as machinery. In addition, when there are multiple sound source objects in the sound space, for any sound source object, other sound source objects may become obstacle objects. Non-sound-emitting objects that do not emit sound, such as building materials or inanimate objects, and sound source objects that emit sound may also become obstacle objects.

[0381] The metadata includes all or part of the information representing the shape of the sound space, the shape information and position information of obstacle objects existing in the sound space, the shape information and position information of sound source objects existing in the sound space, and the position and orientation of the listener in the sound space.

[0382] The sound space may be a closed space or an open space. In addition, the metadata includes information indicating the reflectivity of structures such as floors, walls, or ceilings that can reflect sound in the sound space, and the reflectivity of obstacle objects present in the sound space. Here, the reflectivity is the energy ratio of reflected sound to incident sound, and is set for each frequency band of sound. Of course, the reflectivity can also be set uniformly regardless of the frequency band of the sound. In the case where the sound space is an open space, for example, uniformly set parameters such as attenuation rate, diffracted sound, or initial reflected sound can also be used.

[0383] In the above description, reflectivity is listed as a parameter related to an obstacle object or a sound source object included in metadata, but information other than reflectivity may also be included. For example, information other than reflectivity may also include information related to the material of the object as metadata related to both the sound source object and the non-sound-generating object. Specifically, information other than reflectivity may also include parameters such as diffusivity, transmittance, and sound absorption.

[0384] As information related to the sound source object, it is also possible to include volume, radiation characteristics (directivity), reproduction conditions, the number and type of sound sources emitted from an object, and information specifying the sound source area in the object. For example, the reproduction conditions can also be set to be a sound that flows continuously or a sound triggered by an event. The sound source area in the object can be set by the relative relationship between the position of the listener and the position of the object, or it can be set based on the object. In the case where the sound source area in the object is set by the relative relationship between the position of the listener and the position of the object, the listener can perceive that sound A is emitted from the right side of the object and sound B is emitted from the left side when the listener is looking at it, based on the side where the listener looks at the object. In the case where the sound source area in the object is set based on the object, which sound is emitted from which area of ​​the object regardless of the direction the listener looks at can be fixed. For example, the listener can perceive that when looking at the object from the front, high sound flows from the right side and low sound flows from the left side. In this case, when the listener goes around to the back of the object, the listener can perceive that low sound flows from the right side and high sound flows from the left side when looking from the back.

[0385] Metadata related to the space may include the time until the initial reflected sound, the reverberation time, the ratio of direct sound to diffuse sound, etc. When the ratio of direct sound to diffuse sound is zero, the listener can perceive only the direct sound.

[0386] This embodiment is summarized as follows.

[0387] The sound signal processing method related to the present embodiment includes: an acquisition step of acquiring first position information indicating the position of an object as a moving body in a virtual space, and second position information indicating the position of a listener L in the virtual space; a first calculation step of calculating the moving speed of the object based on the acquired first position information; a second calculation step of calculating the distance between the object and the listener L based on the acquired first position information and the acquired second position information; a generation step of generating an aerodynamic sound signal based on the calculated moving speed and the calculated distance, the aerodynamic sound signal indicating the aerodynamic sound generated by the wind W generated by the movement of the object reaching the ear of the listener L; and an output step of outputting the generated aerodynamic sound signal.

[0388] Thus, the sound signal processing method can generate an aerodynamic sound signal representing the aerodynamic sound generated by the wind W generated by the movement of the object reaching the ears of the listener L based on the position of the object in the virtual space and the position of the listener L, and output the generated aerodynamic sound signal. The aerodynamic sound signal is output to the earphone 200 or the speaker, etc., and the listener L can hear the aerodynamic sound. Therefore, the listener L can perceive that the object is moving and can get a sense of presence. That is, the sound signal processing method of a technical solution related to the present disclosure can give the listener L a sense of presence.

[0389] In the acoustic signal processing method according to the present embodiment, in the generating step, the pneumatic sound signal is generated such that the frequency component is shifted toward the higher frequency side as the calculated moving speed is larger and the volume of the pneumatic sound is increased as the calculated distance is smaller.

[0390] Thus, the aerodynamic sound heard by the listener L can be controlled according to the moving speed of the object and the distance between the object and the listener L, so the listener L can get a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0391] In the acoustic signal processing method according to the present embodiment, the acquisition step acquires availability information indicating whether the subject generates pneumatic sound, and the generation step generates a pneumatic sound signal when the acquired availability information indicates that the subject generates pneumatic sound.

[0392] Thus, the pneumatic sound signal is generated only when the information indicates that the object can generate pneumatic sound. That is, the acoustic signal processing method does not need to generate a pneumatic sound signal when the information does not indicate that the object can generate pneumatic sound, so the processing load for generating pneumatic sound can be suppressed in the acoustic signal processing method.

[0393] Furthermore, the pneumatic sound signal can be generated based on the first position information, the second position information and the availability information. That is, in the acoustic signal processing method according to this embodiment, the pneumatic sound signal can be generated even without information other than the first position information, the second position information and the availability information.

[0394] In the acoustic signal processing method according to the present embodiment, shape information indicating the shape of the object is obtained in the acquisition step, and an aerodynamic sound signal is generated based on the shape indicated by the acquired shape information, the calculated moving speed, and the calculated distance in the generation step.

[0395] Thus, an aerodynamic sound signal is generated according to the shape of the object, the moving speed of the object, and the distance between the object and the listener L, so the listener L can hear the aerodynamic sound with a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0396] Furthermore, an aerodynamic sound signal can be generated based on the first position information, the second position information, and the shape information. That is, in the acoustic signal processing method according to this embodiment, an aerodynamic sound signal can be generated even without information other than the first position information, the second position information, and the shape information.

[0397] In the sound signal processing method related to the present embodiment, in the generation step, the pneumatic sound signal is generated in a manner such that the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, the volume of the pneumatic sound is increased as the calculated distance is smaller, and based on the acquired shape information, the volume of the pneumatic sound is increased as the volume of the object is larger.

[0398] Thus, the aerodynamic sound heard by the listener L can be controlled according to the moving speed of the object, the distance between the object and the listener L, and the volume of the object, so that the listener L can get a better sense of presence. That is, the sound signal processing method can give the listener L a better sense of presence.

[0399] The sound signal processing method related to the present embodiment includes a third calculation step, in which the wind volume per unit time generated when the object moves is calculated based on the calculated moving speed and the obtained shape information, and in the generation step, an aerodynamic sound signal is generated based on the calculated moving speed, the calculated distance and the calculated wind volume.

[0400] Thus, the aerodynamic sound signal is generated according to the moving speed of the object, the distance between the object and the listener L, and the wind volume, so the listener L can hear the aerodynamic sound with a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0401] In the sound signal processing method related to the present embodiment, in the generation step, the pneumatic sound signal is generated in a manner such that the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, the volume of the pneumatic sound is increased as the calculated distance is smaller, and the volume of the pneumatic sound is increased as the calculated wind volume is greater.

[0402] Thus, the aerodynamic sound heard by the listener L can be controlled according to the moving speed of the object, the distance between the object and the listener L, and the wind volume, so the listener L can get a better sense of presence. That is, the sound signal processing method can give the listener L a better sense of presence.

[0403] In the acoustic signal processing method according to the present embodiment, in the generating step, the aerodynamic sound signal is generated based on the calculated moving speed, the calculated distance, and the shape of the human ear.

[0404] Thus, since the aerodynamic sound signal is generated according to the sound generation model based on the shape of the human ear, the listener L can hear the aerodynamic sound with a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0405] Furthermore, by using the ear of an average person as a model, it is unnecessary for a user of a virtual space (for example, a provider of content executed in the virtual space) to prepare data representing aerodynamic sound for the content.

[0406] The acoustic signal processing method according to the present embodiment includes a storage step for storing aerodynamic sound data, wherein the aerodynamic sound data is data obtained by collecting the sound generated by the wind W reaching the human ear or a model simulating the human ear, and in a generation step, aerodynamic sound signals are generated based on the stored aerodynamic sound data, the calculated moving speed and the calculated distance.

[0407] Thus, since the aerodynamic sound signal is generated based on the data collected from the actually generated sound, the listener L can hear the aerodynamic sound with a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0408] In the acoustic signal processing method according to the present embodiment, in the third calculation step, the air volume is calculated based on the area of ​​the object as viewed from the moving direction of the object indicated by the acquired shape information and the calculated moving speed.

[0409] Thus, the air volume is calculated based on the area and moving speed of the object as viewed from the moving direction of the object indicated by the shape information. Furthermore, since the aerodynamic sound signal is generated based on such air volume, the moving speed of the object, and the distance between the object and the listener L, the listener L can listen to the aerodynamic sound with a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0410] In the sound signal processing method related to the present embodiment, in the first calculation step, the moving speed and moving direction of the object are calculated based on the obtained first position information, and in the third calculation step, the wind volume is calculated based on the obtained shape information, the calculated moving direction and the calculated moving speed.

[0411] Thus, the air volume is calculated based on the shape information, the moving direction, and the moving speed. Furthermore, since the aerodynamic sound signal is generated based on the air volume, the moving speed of the object, and the distance between the object and the listener L, the listener L can hear the aerodynamic sound with a better sense of presence. That is, the sound signal processing method can give the listener L a better sense of presence.

[0412] In the sound signal processing method related to the present embodiment, it includes: a generation step, in which a noise signal is processed separately using multiple frequency band emphasis filters, and the processed multiple noise signals are synthesized and processed to generate an aerodynamic sound signal, which represents the aerodynamic sound generated by the wind W reaching the ears of the listener L in the virtual space; and an output step, in which the generated aerodynamic sound signal is output, and the multiple frequency band emphasis filters are filters used to simulate the aerodynamic sound generated by the shape of the listener's ears or head.

[0413] Thus, the sound signal processing method can generate an aerodynamic sound signal representing the aerodynamic sound generated by the wind W reaching the ears of the listener L in the virtual space for the listener L in the virtual space, and output the generated aerodynamic sound signal. The aerodynamic sound signal is output to headphones or speakers, etc., and the listener L can listen to the aerodynamic sound. Therefore, the listener L can perceive that the wind W is blowing in the virtual space and can get a sense of presence. That is, the sound signal processing method can give the listener L a sense of presence.

[0414] Furthermore, the wind W blowing in the virtual space is different from the wind W generated by the movement of the object as described above, and can also be a wind that simulates a breeze or a storm blowing naturally in the real space. Of course, in this case, the position of the source of the wind W is not determined, and the relationship between the position and the position of the listener is not determined. In this case, the listener L can also perceive that the wind W is blowing in the virtual space and can get a sense of presence.

[0415] In the acoustic signal processing method according to the present embodiment, the plurality of band emphasis filters multiply the noise signal processed by the band emphasis filter among the plurality of processed noise signals by a predetermined coefficient corresponding to the band emphasis filter, and in the generation step, synthesize the multiplied signals.

[0416] Thus, the aerodynamic sound generated by the acoustic signal processing method becomes a sound closer to the aerodynamic sound heard by the listener L in the real space. Therefore, the listener L can get a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0417] In the acoustic signal processing method according to the present embodiment, the characteristics of each of the plurality of band emphasis filters vary according to the wind speed of the wind W reaching the ear.

[0418] Thus, the aerodynamic sound generated by the acoustic signal processing method becomes a sound closer to the aerodynamic sound heard by the listener L in the real space. Therefore, the listener L can get a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0419] In the acoustic signal processing method according to the present embodiment, the characteristics of each of the plurality of band emphasis filters vary according to the wind direction of the wind W reaching the ear.

[0420] Thus, the aerodynamic sound generated by the acoustic signal processing method becomes a sound closer to the aerodynamic sound heard by the listener L in the real space. Therefore, the listener L can get a better sense of presence. That is, the acoustic signal processing method can give the listener L a better sense of presence.

[0421] The computer program according to the present embodiment is a computer program for causing a computer to execute the above-described sound signal processing method.

[0422] Thereby, the computer can execute the above-mentioned sound signal processing method according to the computer program.

[0423] The sound signal processing device 100 related to the present embodiment includes: an acquisition unit 110, which acquires first position information indicating the position of an object as a moving body in a virtual space, and second position information indicating the position of a listener L in the virtual space; a first calculation unit 121, which calculates the moving speed of the object based on the acquired first position information; a second calculation unit 122, which calculates the distance between the object and the listener L based on the acquired first position information and the acquired second position information; a generation unit 130, which generates an aerodynamic sound signal based on the calculated moving speed and the calculated distance, the aerodynamic sound signal indicating the aerodynamic sound generated by the wind W generated by the movement of the object reaching the ear of the listener L; and an output unit 140, which outputs the generated aerodynamic sound signal.

[0424] Thus, the sound signal processing device can generate an aerodynamic sound signal representing the aerodynamic sound generated by the wind W generated by the movement of the object reaching the ears of the listener L based on the position of the object in the virtual space and the position of the listener L, and output the generated aerodynamic sound signal. The aerodynamic sound signal is output to the earphone 200 or the speaker, etc., and the listener L can listen to the aerodynamic sound. Therefore, the listener can perceive that the object is moving and can get a sense of presence. That is, the sound signal processing device related to a technical solution of the present disclosure can give the listener L a sense of presence.

[0425] (Other embodiments)

[0426] The above is an explanation of the sound signal processing device 100 and the sound signal processing method of the technical solution of the present disclosure based on the implementation method, but the present disclosure is not limited to the implementation method. For example, other implementation methods that are realized by arbitrarily combining the components described in this specification or excluding some of the components can also be used as implementation methods of the present disclosure. In addition, the above-mentioned implementation methods are implemented by various modifications that can be thought of by those skilled in the art without departing from the main purpose of the present disclosure, that is, without departing from the meaning of the sentences described in the claims. Examples of modifications are also included in the present disclosure.

[0427] In addition, the following aspects are also included in the scope of one or more technical aspects of the present disclosure.

[0428] (1) A part of the components constituting the above-mentioned sound signal processing device 100 may also be a computer system composed of a microprocessor, ROM, RAM, hard disk unit, display unit, keyboard, mouse, etc. A computer program is stored in the RAM or the hard disk unit. The above-mentioned microprocessor operates according to the above-mentioned computer program to achieve its function. Here, the computer program is composed of a plurality of command codes representing instructions to the computer in order to achieve a predetermined function.

[0429] (2) A part of the components constituting the above-mentioned sound signal processing device 100 may be constituted by a system LSI (Large Scale Integration). The system LSI is a super-multifunctional LSI manufactured by integrating multiple components into a single chip. Specifically, it is a computer system composed of a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its function by the microprocessor operating according to the computer program.

[0430] (3) A part of the components constituting the above-mentioned sound signal processing device 100 may also be constituted by an IC card or a single module that is detachable from each device. The above-mentioned IC card or the above-mentioned module is a computer system composed of a microprocessor, ROM, RAM, etc. The above-mentioned IC card or the above-mentioned module may also include the above-mentioned super-multifunctional LSI. The above-mentioned IC card or the above-mentioned module achieves its function by the microprocessor operating according to the computer program. The IC card or the module may also have tamper-resistance.

[0431] (4) In addition, a part of the components constituting the above-mentioned sound signal processing device 100 may be a product in which the above-mentioned computer program or the above-mentioned digital signal is recorded on a recording medium that can be read by a computer, such as a floppy disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. In addition, it may be a digital signal recorded on these recording media.

[0432] Furthermore, a part of the components constituting the sound signal processing device 100 may be obtained by transmitting the computer program or the digital signal via an electric communication line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, or the like.

[0433] (5) The present disclosure may be the methods described above. In addition, the present disclosure may be a computer program that implements the methods on a computer, or a digital signal composed of the computer program.

[0434] (6) Furthermore, the present disclosure may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program and the microprocessor operates according to the computer program.

[0435] (7) In addition, the above-mentioned program or the above-mentioned digital signal may be recorded in the above-mentioned recording medium and transferred, or the above-mentioned program or the above-mentioned digital signal may be transferred via the above-mentioned network, etc., so that it can be implemented by another independent computer system.

[0436] Industrial Applicability

[0437] The present disclosure can be applied to an audio signal processing method and an audio signal processing device, and can be particularly applied to an audio system and the like.

[0438] Description of symbols

[0439] 10, 20 Simulated human head microphone

[0440] 11 Imitation human head

[0441] 12.13 Microphone

[0442] 50, 600 electric fans

[0443] 100. A0001 Audio signal processing device

[0444] 110 Acquisition Department

[0445] 120 Computing Department

[0446] 121 1st Computation Department

[0447] 122 2nd Computation Department

[0448] 123 3rd Computation Department

[0449] 130 Generation Department

[0450] 140 Output

[0451] 150 Storage

[0452] 200 Headphones

[0453] 201 Head sensor unit

[0454] 202 Output

[0455] 300 Display unit

[0456] 400 vehicles

[0457] 500 Server Installation

[0458] 601 Blade

[0459] 700 Aircraft

[0460] 800 round fan

[0461] L Listener

[0462] W Wind

[0463] A0000 Stereo sound reproduction system

[0464] A0002 Sound prompt device

[0465] A0100 Encoding Device

[0466] A0101, A0113 input data

[0467] A0102 Encoder

[0468] A0103 Encoded data

[0469] A0104, A0114 Memory

[0470] A0110, A0130 Decoding Device

[0471] A0111 Sound signal

[0472] A0112 Decoder

[0473] A0120 Encoding device

[0474] A0121 Sending Department

[0475] A0122 Send signal

[0476] A0131 Receiving Department

[0477] A0132 Receive signal

[0478] A0200, A0210 decoder

[0479] A0201, A0211 Space Information Management Department

[0480] A0202 Sound Data Decoder

[0481] A0203, A0213, 900 Rendering Department

[0482] 901 Reverberation Processing Unit

[0483] 902 Initial reflection processing unit

[0484] 903 Distance Attenuation Processing Unit

[0485] 904 Selection Department

[0486] 905 Binaural Processing Unit

Claims

1. A method for processing an acoustic signal, wherein: include: An acquisition step of acquiring first position information indicating a position of an object as a moving body in a virtual space and second position information indicating a position of a listener in the virtual space; a first calculation step of calculating a moving speed of the object based on the first position information obtained; a second calculation step of calculating the distance between the object and the listener based on the first position information obtained and the second position information obtained; a generating step of generating an aerodynamic sound signal based on the calculated moving speed and the calculated distance, the aerodynamic sound signal representing the aerodynamic sound generated by the wind generated by the movement of the object and reaching the ears of the listener; and The output step is to output the generated pneumatic sound signal.

2. The sound signal processing method according to claim 1, wherein: In the generating step, the pneumatic sound signal is generated such that the frequency component is shifted toward a higher frequency side as the calculated moving speed is larger, and the volume of the pneumatic sound is increased as the calculated distance is smaller.

3. The sound signal processing method according to claim 1, wherein: In the acquisition step, the information indicating whether the object generates the aerodynamic sound is acquired. In the generating step, the pneumatic sound signal is generated when the acquired ability information indicates that the object can generate the pneumatic sound.

4. The sound signal processing method according to claim 1, wherein: In the acquisition step, shape information indicating the shape of the object is acquired. In the generating step, the pneumatic sound signal is generated based on the shape indicated by the acquired shape information, the calculated moving speed, and the calculated distance.

5. The sound signal processing method according to claim 4, wherein: In the above-mentioned generation step, the above-mentioned pneumatic sound signal is generated in a manner such that the frequency component is shifted toward the high frequency side as the calculated moving speed is greater, the volume of the above-mentioned pneumatic sound is increased as the calculated distance is smaller, and the volume of the above-mentioned pneumatic sound is increased as the volume of the above-mentioned object is greater based on the above-mentioned shape information obtained.

6. The sound signal processing method according to claim 4, wherein: The method comprises a third calculation step of calculating the wind volume per unit time generated when the object moves based on the calculated moving speed and the acquired shape information, In the generating step, the aerodynamic sound signal is generated based on the calculated moving speed, the calculated distance, and the calculated air volume.

7. The sound signal processing method according to claim 6, wherein: In the above-mentioned generation step, the above-mentioned pneumatic sound signal is generated in a manner that the frequency component shifts toward the high frequency side as the calculated moving speed is greater, the volume of the above-mentioned pneumatic sound is greater as the calculated distance is smaller, and the volume of the above-mentioned pneumatic sound is greater as the calculated wind volume is greater.

8. The sound signal processing method according to claim 1, wherein: In the generating step, the aerodynamic sound signal is generated based on the calculated moving speed, the calculated distance, and the shape of a human ear.

9. The sound signal processing method according to claim 1, wherein: The method comprises a step of storing aerodynamic sound data, wherein the aerodynamic sound data is data obtained by collecting the sound generated by the wind reaching the human ear or a model simulating the human ear, In the generating step, the pneumatic sound signal is generated based on the stored pneumatic sound data, the calculated moving speed, and the calculated distance.

10. The sound signal processing method according to claim 6, wherein: In the third calculation step, the air volume is calculated based on the area of ​​the object as viewed from the moving direction of the object indicated by the acquired shape information and the calculated moving speed.

11. The sound signal processing method according to claim 6, wherein: In the first calculation step, the moving speed and moving direction of the object are calculated based on the first position information obtained. In the third calculation step, the air volume is calculated based on the acquired shape information, the calculated moving direction, and the calculated moving speed.

12. A method for processing an audio signal, wherein: include: a generating step of processing the noise signal using a plurality of frequency band emphasis filters respectively, synthesizing the processed plurality of noise signals, thereby generating an aerodynamic sound signal representing the aerodynamic sound generated by the wind reaching the ears of the listener in the virtual space; and an output step of outputting the generated pneumatic sound signal, The plurality of frequency band emphasis filters are filters for simulating the aerodynamic sound generated by the shape of the ear or head of the listener.

13. The sound signal processing method according to claim 12, wherein: The plurality of band emphasis filters multiply the noise signal processed by the band emphasis filter among the plurality of noise signals processed by the noise signal by a predetermined coefficient corresponding to the band emphasis filter. In the above-mentioned generation step, after the above-mentioned multiplication is performed, the above-mentioned synthesis is performed.

14. The sound signal processing method according to claim 12, wherein: The characteristics of each of the plurality of band emphasis filters vary according to the wind speed of the wind reaching the ear.

15. The sound signal processing method according to claim 12, wherein: The characteristics of each of the plurality of band emphasis filters vary according to the wind direction of the wind reaching the ear.

16. A program for causing a computer to execute the sound signal processing method according to any one of claims 1 to 15.

17. An audio signal processing device, wherein: have: an acquisition unit that acquires first position information indicating a position of an object as a moving body in a virtual space and second position information indicating a position of a listener in the virtual space; a first calculation unit that calculates a moving speed of the object based on the acquired first position information; a second calculation unit that calculates a distance between the object and the listener based on the first position information obtained and the second position information obtained; a generating unit that generates an aerodynamic sound signal representing aerodynamic sound generated by wind generated by the movement of the object and reaching the ears of the listener based on the calculated moving speed and the calculated distance; and The output unit outputs the generated pneumatic sound signal.

Images