Information processing apparatus, information processing method, and program

Abstract

An information processing method is an information processing method executed by a computer, including the steps of: a step (S101) of acquiring sound information including an audio signal and information of a position of a sound source object in a three-dimensional sound field; a step (S107) of generating a first sound signal using a head-related transfer function corresponding to a direction of arrival based on the position of the sound source object and a position of a user in the three-dimensional sound field and the audio signal; a step (S106) of generating a second sound signal using a head-related transfer function corresponding to a representative direction based on a position of a representative point set in the three-dimensional sound field and the position of the user and the audio signal; and a step (S109) of generating an output sound signal by synthesizing the generated first sound signal and the second sound signal to generate the output sound signal.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing methods, and procedures. Background Technology

[0002] Previously, there were known technologies related to sound reproduction that allowed users (also known as listeners of the playing sound) to perceive stereo sound within a virtual three-dimensional space (see, for example, Patent Document 1). Furthermore, in order for the user to perceive the sound as if it were coming from a sound source object within this three-dimensional space, processing is required to generate output sound information from the original sound information. In particular, a large amount of processing is required to reproduce stereo sound corresponding to the user's body movements within the virtual space. With the development of computer graphics (CG), it is relatively easy to construct visually complex virtual environments, and technologies for realizing corresponding auditory information have become increasingly important. Furthermore, when processing is performed in advance to generate output sound information from the sound information, a large storage area is needed to store the pre-calculated processing results. In addition, transmitting such large processing result data may require a wide communication bandwidth.

[0003] To achieve a more realistic sound environment, the number of objects emitting sound in the virtual 3D space increases, as does the amount of secondary sound based on acoustic effects such as reflected sound, diffraction, and reverberation. Furthermore, these secondary sounds need to be appropriately altered according to the user's movement, thus requiring extensive processing. Therefore, to reduce this processing load, a transformation technique called panning is known. This technique represents sound in 3D space using sound from several pre-defined representative points.

[0004] (Existing technical documents) (Patent Documents) Patent Document 1: Japanese Patent Application Publication No. 2020-18620 Summary of the Invention

[0005] The problem that the invention aims to solve However, in transformation processes such as audio-visual processing, sound quality degradation can occur as a trade-off with reducing processing volume. Therefore, the object of this disclosure is to provide an information processing method for appropriately performing transformation processing.

[0006] Methods for solving problems The information processing method disclosed herein, executed by a computer, includes the following steps: acquiring sound information, the sound information including an audio signal and information about the location of a sound source object within a three-dimensional sound field; generating a first sound signal using a head-related transfer function corresponding to the location of the sound source object and the direction of arrival of the user's location within the three-dimensional sound field, along with the audio signal; generating a second sound signal using a head-related transfer function corresponding to the location of a representative point set within the three-dimensional sound field and the direction of arrival of the user's location, along with the audio signal; and generating an output sound signal by synthesizing the generated first sound signal and the second sound signal to generate the output sound signal.

[0007] Furthermore, one aspect of the information processing apparatus disclosed herein includes: an acquisition unit for acquiring sound information, the sound information including an audio signal and information about the position of a sound source object within a three-dimensional sound field; a first generation unit for generating a first sound signal using a head-related transfer function corresponding to the position of the sound source object and the direction of arrival of a user's position within the three-dimensional sound field, and the audio signal; a second generation unit for generating a second sound signal using a head-related transfer function corresponding to the position of a representative point set within the three-dimensional sound field and the direction of arrival of a user's position, and the audio signal; and a synthesis unit for synthesizing the generated first and second sound signals to generate an output sound signal.

[0008] Furthermore, one aspect of this disclosure can also be implemented as a program for causing a computer to perform the aforementioned information processing method.

[0009] Furthermore, these general or specific methods can also be implemented by non-transitory recording media such as systems, devices, methods, integrated circuits, computer programs, or computer-readable CD-ROMs, and can also be implemented by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0010] Invention Effects This disclosure enables appropriate transformation processing. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating an example of the use of the sound reproduction system according to the embodiment.

[0012] Figure 2 This is a block diagram illustrating the functional structure of the sound reproduction system according to the embodiment.

[0013] Figure 3This is a diagram used to illustrate an example of a sound signal involved in the implementation method.

[0014] Figure 4 This is a block diagram illustrating the functional structure of the acquisition unit according to the embodiment.

[0015] Figure 5 This is a block diagram illustrating the functional structure of the output sound generation unit according to the embodiment.

[0016] Figure 6 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0017] Figure 7 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0018] Figure 8 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0019] Figure 9 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0020] Figure 10 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0021] Figure 11 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0022] Figure 12 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0023] Figure 13 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0024] Figure 14 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0025] Figure 15 This is a diagram illustrating another example of the sound reproduction system involved in the implementation method.

[0026] Figure 16 This is a flowchart illustrating an example of the operation of the information processing apparatus according to the embodiment. Detailed Implementation

[0027] (Insights that become the basis for public disclosure) Previously, there were known technologies for sound reproduction that allowed users to perceive stereo sound within a virtual three-dimensional space (hereinafter sometimes referred to as a three-dimensional sound field) (see, for example, Patent Document 1). Using this technology, users could perceive sound as if the sound source object existed at a predetermined location within the virtual space, and that the sound was coming from that direction. To locate the sound image at such a predetermined location within the virtual three-dimensional space, computational processing was required, for example, to generate, for the signal of the sound being emitted by the sound source object (also referred to as the sound emitted at the sound source object or the played sound), the interaural time difference of sound arrival, the interaural sound level difference (or sound pressure difference), etc., which could be perceived as stereo sound. This computational processing was performed by applying a three-dimensional sound filter. A three-dimensional sound filter is a filter used for information processing; when this filter is applied to the original sound information, the reproduced output sound signal allows for a stereo perception of the sound's direction, distance, location, sound source size, spatial breadth, etc.

[0028] As an example of computational processing in this application of three-dimensional acoustic filters, a process is known to convolve the head correlation transfer function (HRF) used to perceive sound as coming from a specified direction with the signal of the target sound. By performing this HRF convolution at a sufficiently small angle, targeting the direction of arrival of the played sound from the sound source object's location to the user's location, the sense of presence experienced by the user can be improved.

[0029] Furthermore, in recent years, the development of virtual reality (VR) technology has been actively underway. A key aspect of VR is that the position of a sound source object within a virtual 3D space can appropriately change according to the user's movement, allowing the user to experience the feeling of moving within the virtual space. This requires relatively shifting the position of the sound image within the virtual space based on the user's movement. This processing has traditionally been done by applying 3D acoustic filters, such as the head-related transfer function, to the raw sound information. However, when the user moves within 3D space, the sound propagation path constantly changes due to sound reverberation and interference, affecting the positional relationship between the sound source object and the user. Therefore, it is necessary to determine the sound propagation path from the sound source object each time based on the positional relationship between the sound source object and the user, and to convolve the transfer function considering sound reverberation and interference. However, this information processing is massive, and without large-scale processing equipment, a significant improvement in the sense of presence cannot be expected.

[0030] Therefore, to reduce this ever-increasing processing load, an attempt is being made to apply panning to the playback sound to reduce the amount of convolution required by the head correlation transfer function. Specifically, instead of performing head correlation transfer function convolution on the playback sound of each of multiple sound source objects in three-dimensional space, the playback sound from the sound source objects is re-represented by sounds from several pre-defined representative points (representative tones) in three-dimensional space. Then, by convolving the head correlation transfer function from the representative point to the user's position with the representative tones, a stereo sound comparable to that perceived by the user can be achieved. If the number of representative points is less than the number of original sound source objects, the number of objects to be convolved by the head correlation transfer function will naturally decrease, thus being advantageous from a processing load perspective.

[0031] However, when applying this type of image processing, it is sometimes inappropriate. Specifically, by applying image processing, the sound changes compared to the original playback sound, which can sometimes lead to a decrease in sound quality (sound degradation). Therefore, in this disclosure, image processing is applied to some of the audio signals contained in the sound information, while for other parts, image processing is not applied, but rather a transfer function convolution is performed based on the positional relationship between the sound source object and the user. Thus, it is possible to reduce the processing load on the audio signals to which image processing is applied, while suppressing sound degradation in other parts. In other words, from the viewpoint of suppressing sound degradation, the transformation processing (image processing) can be performed more appropriately.

[0032] A more specific summary of this disclosure is shown below.

[0033] The information processing method disclosed in the first aspect is a computer-executed information processing method, comprising the following steps: acquiring sound information, the sound information including an audio signal and information about the location of a sound source object within a three-dimensional sound field; generating a first sound signal by using a head-related transfer function corresponding to the location of the sound source object and the direction of arrival of the user's location within the three-dimensional sound field, and the audio signal; generating a second sound signal by using a head-related transfer function corresponding to the location of a representative point set within the three-dimensional sound field and the direction of arrival of the user's location, and the audio signal; and generating an output sound signal by synthesizing the generated first sound signal and the second sound signal to generate the output sound signal.

[0034] This information processing device can perform the following operations: using a first generation unit, a first sound signal is generated using a head-related transfer function corresponding to the calculated direction of arrival; using a second generation unit, a second sound signal is generated using a head-related transfer function corresponding to the representative direction; and the first and second sound signals are synthesized to generate an output sound signal. For example, when generating an output sound signal from sound information, convolution with the head-related transfer function corresponding to the representative direction can be performed in part to reduce processing load, while convolution with the head-related transfer function corresponding to the direction of arrival can be performed in other parts to suppress sound degradation. In this way, a balance can be achieved between reducing processing load and suppressing sound degradation, thus enabling appropriate transformation processing.

[0035] Furthermore, the information processing method involved in the second method is based on the information processing method described in the first method. In the step of generating the first sound signal, the first sound signal is generated by convolving the head-related transfer function corresponding to the direction of arrival with the playback sound emitted by the sound source object through the audio signal. In the step of generating the second sound signal, a transformation process is performed to transform the playback sound into a representative sound transmitted from the representative point, and the second sound signal is generated by convolving the head-related transfer function corresponding to the representative direction with the representative sound.

[0036] Accordingly, the first generation unit generates a first output sound signal by convolving the head-related transfer function corresponding to the direction of arrival with the played sound. The second generation unit generates a second sound signal that represents the sound from the direction of arrival by performing transformation processing such as sound imaging processing, using representative sounds from representative points set within the three-dimensional sound field. These signals are then synthesized to generate the output sound signal. For example, the second generation unit can be used for parts where transformation processing can effectively reduce the processing load, while the first generation unit can be used for parts where transformation processing easily leads to sound degradation.

[0037] Furthermore, the information processing method involved in the third method is based on the information processing method described in the second method, wherein, in the transformation process, the playback sound is transformed into the representative sound by applying time offset adjustment and gain adjustment.

[0038] Therefore, in the transformation process, the played sound can be transformed into a representative sound by applying time offset and gain adjustments. In this way, even with transformation processing, the sense of incongruity can be reduced, thereby generating an output sound signal with a higher sense of presence.

[0039] Furthermore, the information processing method involved in the fourth method is based on the information processing method described in any of the first to third methods, and the sound information includes a plurality of audio signals and the positions of a plurality of sound source objects corresponding one-to-one with the plurality of audio signals. The information processing method further includes a decision-making step, which, for each of the plurality of audio signals contained in the sound information, determines whether to use it to generate the first sound signal or to use it to generate the second sound signal.

[0040] Accordingly, it is possible to determine, for the audio signals corresponding to each of the multiple sound source objects contained in the sound information, that is, for the sound emitted by each sound source object, whether to perform convolution with the head correlation transfer function corresponding to the direction of arrival in order to reduce the amount of processing, or to perform convolution with the head correlation transfer function corresponding to the direction of arrival in order to suppress sound degradation.

[0041] Furthermore, the information processing method involved in the fifth method is based on the information processing method described in any of the first to third methods. The sound information includes a plurality of audio signals and the location of a sound source object that corresponds to the plurality of audio signals in a many-to-one manner. The information processing method further includes a decision step of determining whether to use each of the plurality of audio signals included in the sound information to generate the first sound signal or to generate the second sound signal.

[0042] Accordingly, it is possible to determine whether to perform convolution with the head correlation transfer function corresponding to the direction of arrival to reduce the amount of processing, or to perform convolution with the head correlation transfer function corresponding to the direction of arrival to suppress sound degradation, for multiple audio signals corresponding to a sound source object contained in the sound information, i.e., multiple sounds emitted by a sound source object.

[0043] Furthermore, the information processing method involved in the sixth method is based on the information processing method described in the fifth method, and the information processing method further includes a step of transforming the acquired sound information, wherein the acquired sound information is transformed into sound information including multiple audio signals.

[0044] Therefore, by transforming the acquired sound information, it is possible to obtain sound information containing multiple audio signals.

[0045] Furthermore, the information processing method involved in the seventh method is based on the information processing method described in any of the fourth to sixth methods. The plurality of audio signals include audio signals associated with direct sound and audio signals associated with secondary sound. The direct sound is the sound that arrives directly at the user's location from the location of the corresponding sound source object. The secondary sound is the sound that accompanies the direct sound and arrives via a different path than the direct sound. In the decision-making step, it is decided to use the audio signal associated with the direct sound for the generation of the first sound signal, and it is decided to use the audio signal associated with the secondary sound for the generation of the second sound signal.

[0046] Accordingly, it is possible to convolve the secondary sound with the head-related transfer function corresponding to the direction of arrival to reduce the amount of processing, while convolving the direct sound with the head-related transfer function corresponding to the direction of arrival to suppress sound degradation.

[0047] Furthermore, the information processing method involved in the eighth method is based on the information processing method described in any of the fourth to seventh methods. The plurality of audio signals include audio signals associated with direct sound and audio signals associated with secondary sound. The direct sound is the sound that arrives directly from the location of the corresponding sound source object to the location of the user. The secondary sound is the sound that accompanies the direct sound and arrives along a different path than the direct sound. The secondary sound includes another secondary sound that is further generated accompanying one of the secondary sounds. In the decision-making step, for each audio signal associated with the secondary sound, a decision is made on whether to use it for the generation of the second sound signal based on information in each of the secondary sounds related to the generation system from the direct sound.

[0048] Accordingly, for each audio signal associated with a secondary sound, it is possible to determine whether to use it to generate a first sound signal or a second sound signal, based on information related to the generation system from the direct sound in each secondary sound.

[0049] Furthermore, the information processing method involved in the ninth method is based on the information processing method described in any of the fourth to eighth methods. In the step of making a decision, for each of the plurality of audio signals, a decision is made on whether to use it for generating the first sound signal or for generating the second sound signal based on the direction of the corresponding sound source object relative to the frontal direction of the user.

[0050] Accordingly, for each of the multiple audio signals, it is possible to determine whether to use it to generate the first sound signal or the second sound signal, based on the direction of the corresponding sound source object relative to the user's frontal direction.

[0051] Furthermore, the information processing method involved in the 10th method is based on the information processing method described in any of the 4th to 9th methods. In the decision-making step, for each of the plurality of audio signals, a decision is made on whether to use it for the generation of the first audio signal or for the generation of the second audio signal based on the distance from the user's location to the location of the corresponding sound source object.

[0052] Accordingly, for each of the multiple audio signals, it is possible to determine whether to use it to generate a first sound signal or a second sound signal, based on the distance from the user's position to the position of the corresponding sound source object.

[0053] Furthermore, the information processing method involved in the 11th method is based on the information processing method described in any of the 4th to 10th methods. In the step of making a decision, it is determined whether to use it for generating the first sound signal or for generating the second sound signal based on the characteristics of each of the plurality of audio signals related to the sound quality degradation when generating the second sound signal.

[0054] Accordingly, for each of the multiple audio signals, it is possible to determine whether to use it to generate the first audio signal or the second audio signal based on the characteristics related to sound quality degradation when generating the second audio signal.

[0055] Furthermore, the information processing method involved in the 12th method is based on the information processing method described in any of the 1st to 11th methods. The sound information includes the audio signal associated with the direct sound and the audio signal associated with the secondary sound. The direct sound is the sound that arrives directly from the location of the corresponding sound source object to the location of the user. The secondary sound is the sound that accompanies the direct sound and arrives along a different path than the direct sound. When the audio signal associated with the secondary sound is identified as reverberant based on information related to the system that generates the direct sound, it is determined not to be used for the generation of the second sound signal.

[0056] Accordingly, it is possible to determine, for each of the audio signals relating to secondary sounds, that the secondary sound identified as reverberant sound will not be used in the generation of the second sound signal.

[0057] Furthermore, the information processing apparatus according to the 13th method comprises: an acquisition unit for acquiring sound information, the sound information including an audio signal and information about the position of a sound source object within a three-dimensional sound field; a first generation unit for generating a first sound signal using a head-related transfer function corresponding to the position of the sound source object and the direction of arrival of the user's position within the three-dimensional sound field, and the audio signal; a second generation unit for generating a second sound signal using a head-related transfer function corresponding to the position of a representative point set within the three-dimensional sound field and the direction of arrival of the user's position, and the audio signal; and a synthesis unit for synthesizing the generated first and second sound signals to generate an output sound signal.

[0058] Therefore, the same effect as the information processing method described above can be achieved.

[0059] Furthermore, the program involved in method 14 is a program used to cause a computer to perform the aforementioned information processing method.

[0060] Therefore, it is possible to use a computer to achieve the same effect as the information processing method described above.

[0061] Furthermore, these general or specific methods can be implemented by systems, devices, methods, integrated circuits, computer programs, or non-transitory recording media such as computer-readable CD-ROMs, and can also be implemented by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0062] The embodiments will now be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below are all illustrative generalizations or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement and connection of constituent elements, steps, and the order of steps shown in the following embodiments are all examples and are not intended to limit this disclosure. Furthermore, in the constituent elements of the following embodiments, constituent elements not described in the independent technical solutions will be described as arbitrary constituent elements. In addition, all figures are schematic diagrams and are not necessarily strict illustrations. Furthermore, in the figures, substantially identical structures will be labeled with the same reference numerals, and repeated descriptions will be omitted or simplified.

[0063] Furthermore, in the following description, elements may be assigned ordinal numbers such as first, second, third, etc. These ordinal numbers are assigned to identify the elements and do not necessarily correspond to a meaningful order. These ordinal numbers may be interchanged, added, or removed as appropriate.

[0064] Furthermore, while the following description refers to audio signals contained in sound information, audio signals may also be represented as speech signals or sound signals. In other words, in this disclosure, audio signals have the same meaning as speech signals or sound signals.

[0065] (Implementation Method) [summary] First, an overview of the sound reproduction system involved in the implementation method will be described. Figure 1 This is a schematic diagram illustrating an example of the use of the sound reproduction system according to the embodiment. Figure 1 The image shows a user 99 using the sound reproduction system 100.

[0066] Figure 1 The sound reproduction system 100 shown is used simultaneously with, for example, the stereoscopic image reproduction device 300. By simultaneously viewing and listening to stereoscopic images and stereoscopic sounds, the images enhance the auditory sense of presence, and the sounds enhance the visual sense of presence; the two complement each other, allowing the user to experience as if they were at the scene where the images and sounds were filmed. For example, it is known that when displaying an image of people conversing (moving images), even if the location of the sound image (sound source object) is deviated from the person's mouth, the user will still perceive it as conversation emanating from that person's mouth. By combining images and sounds in this way, such as correcting the position of the sound image through visual information, the sense of presence is enhanced.

[0067] The stereoscopic image reproduction device 300 is an image display device worn on the head of the user 99. Therefore, the stereoscopic image reproduction device 300 moves as a unit with the user 99's head. For example, as shown in the figure, the stereoscopic image reproduction device 300 is a glasses-type device supported by the user 99's ears and nose.

[0068] The stereoscopic image reproduction device 300 alters the displayed image based on the movement of the user's head (99), thus giving the user the perception that their head is moving within a three-dimensional image space. In other words, when an object in the three-dimensional image space is in front of the user (99), if the user (99) looks to the right, the object moves to the user's left; if the user (99) looks to the left, the object moves to the user's right. In this way, the stereoscopic image reproduction device 300, relative to the user (99), moves the three-dimensional image space in the opposite direction to the user's movement.

[0069] The stereoscopic image reproduction device 300 displays two images with parallax for the left and right eyes of the user 99, respectively. The user 99 can perceive the three-dimensional position of objects in the displayed images based on the parallax. Furthermore, when the user 99 closes their eyes, such as when the sound reproduction system 100 is used to play soothing music for sleep, the stereoscopic image reproduction device 300 does not need to be used simultaneously. In other words, the stereoscopic image reproduction device 300 is not an essential component of this disclosure. Besides dedicated image display devices, the stereoscopic image reproduction device 300 can also be a common portable terminal owned by the user 99, such as a smartphone or tablet.

[0070] In addition to a display screen for showing images, this universal portable terminal is equipped with various sensors for detecting the terminal's posture and motion. Furthermore, it includes a processor for information processing and can connect to network and cloud server devices to send and receive information. In other words, the stereoscopic image reproduction device 300 and the audio reproduction system 100 can also be realized through a combination of a smartphone and a general-purpose headset without information processing capabilities.

[0071] As shown in this example, the head movement detection function, image prompting function, image information processing function for prompting, sound prompting function, and sound information processing function for prompting can be appropriately configured in one or more devices to realize the stereoscopic image reproduction device 300 and the sound reproduction system 100. When the stereoscopic image reproduction device 300 is not required, it is sufficient to appropriately configure the head movement detection function, sound prompting function, and sound information processing function for prompting in one or more devices. For example, the sound reproduction system 100 can be realized using a processing device such as a computer or smartphone with sound information processing function for prompting, and headphones with head movement detection function and sound prompting function.

[0072] The sound reproduction system 100 is an audio prompting device worn on the head of the user 99. Therefore, the sound reproduction system 100 moves as a unit with the user 99's head. For example, the sound reproduction system 100 in this embodiment is a so-called over-ear headphone type device. Furthermore, there is no particular limitation on the form of the sound reproduction system 100; for example, it can be two earbud-type devices worn independently on the left and right ears of the user 99.

[0073] The sound reproduction system 100 alters the audible prompts based on the movement of the user 99's head, making the user 99 perceive as if they are moving their head within a three-dimensional sound field. Therefore, as described above, the sound reproduction system 100, relative to the user 99's movement, shifts the three-dimensional sound field in the opposite direction to the user 99's movement.

[0074] Here, when user 99 moves within the three-dimensional sound field, the relative position of the sound source object with respect to user 99's position within the three-dimensional sound field changes. Therefore, whenever user 99 moves, calculations based on the positions of the sound source object and user 99 are required to generate an output sound signal for reproduction. This processing is typically very complex; therefore, in this disclosure, from the viewpoint of reducing processing complexity, image processing, as a transformation process, is applied to represent the played sound using a representative sound from a representative point. As a result, user 99 can perceive the played sound from the sound source object simply by convolving the head-related transfer function with the representative sound. On the other hand, when image processing is applied, the sound localization is impaired, and the position of the sound image may become difficult to perceive (blurred). Therefore, applying image processing to sounds that are significantly affected by impaired localization also has drawbacks such as damaging the original sense of presence. Therefore, in this disclosure, image processing is applied to a portion of the audio signal, while image processing is not applied to other portions to generate the output sound signal, thereby forming a three-dimensional sound field. In this embodiment, we will describe an example of using image processing as a transformation process. However, the transformation process is not limited to image processing. Any transformation process can be applied as long as it is expected to reduce the amount of processing based on the conditions. Furthermore, specific examples of image processing will be given for explanation. However, image processing is not limited to the methods described in the specific examples below. Existing image processing methods such as VBAP (vector-based amplitude translation), DBAP (distance-based amplitude translation), and Ambisonics (omnidirectional three-dimensional sound field technology) can also be applied.

[0075] [constitute] Next, refer to Figure 2 The configuration of the sound reproduction system 100 of this embodiment will be described. Figure 2 This is a block diagram illustrating the functional structure of the sound reproduction system according to the embodiment.

[0076] like Figure 2 As shown, the sound reproduction system 100 of this embodiment includes an information processing device 101, a communication module 102, a detector 103, a driver 104, and a database 105.

[0077] The information processing device 101 is a computing device used to perform various signal processing in the sound reproduction system 100. The information processing device 101 is implemented, for example, by a computer equipped with a processor and memory, and the program stored in the memory is executed by the processor. By executing this program, functions related to the various functional units described below are performed.

[0078] The information processing device 101 includes an acquisition unit 111, a path calculation unit 121, an output tone generation unit 131, and a signal output unit 141. Details of each functional unit of the information processing device 101 will be described below along with details of the structure other than the information processing device 101.

[0079] The communication module 102 is an interface device for receiving sound information input to the sound reproduction system 100. The communication module 102 includes, for example, an antenna and a signal converter, and receives sound information from an external device via wireless communication. More specifically, the communication module 102 uses the antenna to receive a wireless signal representing sound information converted into a wireless communication format, and uses the signal converter to convert the wireless signal back into sound information. Thus, the sound reproduction system 100 acquires sound information from an external device via wireless communication. The sound information acquired by the communication module 102 is then acquired by the acquisition unit 111. As described above, the acquisition unit 111 is an example of a sound acquisition unit. The sound information is input to the information processing device 101 in the above manner. Furthermore, communication between the sound reproduction system 100 and the external device can also be performed via wired communication.

[0080] The sound information acquired by the sound reproduction system 100 is encoded, for example, in a format specified by MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound information includes information about the playback sound reproduced by the sound reproduction system 100, as well as location-related information regarding the position of the sound image within the three-dimensional sound field (i.e., perceived as sound coming from a specified direction). The sound information can also be interpreted as information related to the sound source object. That is, the sound information includes the position of the sound source object within the three-dimensional sound field, and the sound emitted by the sound source object. Furthermore, the sound information may contain flags used to determine whether sound image processing is applied. These flags will be described in detail later.

[0081] As described above, sound information is obtained as input data, including information about the played sound, i.e., the sound signal (audio signal), and information about the location of the sound source object within the three-dimensional sound field, as other information. This other information may also include information used to define the three-dimensional sound field. Therefore, information including the location of the sound source object and other information used to define the three-dimensional sound field can be collectively referred to as space-related information (spatial information). When viewed primarily as an audio signal, the input data can be said to be sound information with additional information (metadata) attached to it. Furthermore, when viewed primarily as spatial information, the input data can be said to be information about the spatial information with the audio signal attached. Alternatively, due to these two aspects of the input data, it can also be considered as spatial sound information.

[0082] As a concrete example, the sound information contains information related to multiple sounds, including a first playback sound and a second playback sound, and perceives the sound image of each sound reproduction as sound coming from different positions within a three-dimensional sound field. Therefore, the sound source object of the first playback sound is located at a first position within the three-dimensional sound field, and the sound source object of the second playback sound is located at a second position within the three-dimensional sound field. As described above, the sound information contains multiple sounds. That is, the sound information contains multiple sound signals corresponding to the first playback sound and the second playback sound, respectively, and the positions of multiple sound source objects at the first and second positions corresponding one-to-one with these multiple sound signals.

[0083] Figure 3 This is a diagram used to illustrate an example of a sound signal involved in the implementation method. For example, such as... Figure 3 As shown in (a), the sound information sometimes includes a first direct sound signal arriving at the user's position 99 from a first position (from a first direction) and a second direct sound signal arriving at the user's position 99 from a second position (from a second direction). Furthermore, the newly acquired sound information may only contain information related to the playback sound. In this case, it can be configured to acquire additional information related to the specified position and prepare all such information before proceeding with subsequent processing. Furthermore, as described above, the sound information includes first sound information related to the first playback sound and second sound information related to the second playback sound, but it is also possible to acquire multiple sound information containing these information separately and reproduce them simultaneously (i.e., process them as a single sound information), thereby locating the sound image at different positions within the three-dimensional sound field, causing the playback sound to come from different directions.

[0084] Alternatively, sound information may sometimes include multiple sound signals and the location of a sound source object that corresponds to each of the multiple sound signals in a many-to-one relationship. For example, this sound information may be used when a sound source object emits multiple playback sounds. For example, each of the multiple sound signals may correspond to a direct sound that travels directly from the location of the sound source object to the user's location, and a secondary sound (sound generated through indirect propagation) that accompanies the direct sound and arrives via a different path than the direct sound.

[0085] For example, such as Figure 3As shown in (b), the newly acquired sound information contains sound signals related to the direct sound. Through the transformation processing of secondary sounds, this information is transformed into sound information containing individual sound signals such as reverberation, first reflection, and diffraction. In calculating this transformation processing of secondary sounds, information about the spatial environment conditions of the three-dimensional sound field (e.g., the position of objects within the three-dimensional sound field, reflection, diffraction characteristics, etc.) is used. In this way, secondary sounds are calculated from sound information related to a played sound based on the spatial environment conditions of the three-dimensional sound field, and therefore are not included in the newly acquired sound information. Instead, sound information containing these secondary sounds is generated through the transformation processing of secondary sounds. The propagation of one secondary sound may sometimes further generate another secondary sound. Furthermore, information about the spatial environment conditions is part of the spatial information and can be acquired along with the input sound information and sound signal. Alternatively, the sound signal and spatial information can be acquired separately. That is, the sound information can be acquired from a single file or bitstream, or it can be acquired separately from multiple files or bitstreams. For example, the sound signal and spatial information can be acquired from different files or bitstreams, or each can be acquired from multiple files or bitstreams.

[0086] exist Figure 3 In example (b), a secondary reflection is generated from a primary reflection. As shown, these secondary sounds are labeled with identifiers of their relationships (e.g., parent, child, grandchild, etc.) as information related to the genealogy (in other words, the generation system) of the generation relationship from the primary sound. Here, the information related to the generation system can be understood as information assigned to each sound to identify the type of sound based on the generation system. For example, if the primary sound is the parent, then the primary reflection is the child, and the secondary reflection is the grandchild. Conversely, if the secondary reflection is the grandchild, then the parent sound contains the primary sound, and the child sound contains the primary reflection. In this way, the information related to the generation system can be used to identify the type of sound. That is, the information related to the generation system is used to identify which of several types the sound belongs to, such as primary sound, primary reflection, secondary reflection, reverberation, and diffraction. Alternatively, when the primary sound is set to generation 0, it can be numerically represented as the first generation to which the primary reflection belongs, the second generation to which the secondary reflection belongs, and so on. This information about the generation system is used to determine whether to set the audio signal for application of image processing. Furthermore, the number of generations allowed to be generated, starting from the direct sound, can also be set according to the scale of computing resources.

[0087] In this way, there is no particular limitation on the form of the input sound information, as long as the sound reproduction system 100 has an acquisition unit 111 that can handle various forms of sound information. Before or after input, the sound information becomes a state containing multiple audio signals, so as to at least distinguish between audio signals that have undergone sound imaging processing and audio signals that have not undergone sound imaging processing.

[0088] Here, use Figure 4 This describes an example of the acquisition section 111. Figure 4 This is a block diagram illustrating the functional structure of the acquisition unit according to an embodiment. For example... Figure 4 As shown, the acquisition unit 111 in this embodiment includes, for example, an encoded audio information input unit 112, a decoding processing unit 113, and a sensing information input unit 114.

[0089] The encoded sound information input unit 112 is a processing unit that inputs encoded (in other words, the encoded) sound information acquired by the acquisition unit 111. The encoded sound information input unit 112 outputs the input sound information to the decoding processing unit 113. The decoding processing unit 113 is a processing unit that decodes (in other words, decodes) the sound information output from the encoded sound information input unit 112 to generate the playback tone and sound source object position contained in the sound information in a form used in subsequent processing. The function of the sensing information input unit 114, along with that of the detector 103, will be described below.

[0090] Detector 103 is a device for detecting the movement speed of the user 99's head. Detector 103 is composed of a combination of various sensors used for motion detection, such as a gyroscope sensor and an accelerometer. In this embodiment, detector 103 is built into the sound reproduction system 100, but it can also be built into an external device, such as a stereoscopic image reproduction device 300 that moves according to the movement of the user 99's head, just like the sound reproduction system 100. In this case, detector 103 may not be included in the sound reproduction system 100. Furthermore, as detector 103, an external camera device or the like can be used to capture images of the moving user 99, and the movement of the user 99 can be detected by processing the captured images.

[0091] The detector 103 is integrally fixed to the housing of the sound reproduction system 100, for example, and detects the movement speed of the housing. After the sound reproduction system 100, including the housing, is worn by the user 99, it moves together with the user 99's head, so the detector 103 can ultimately detect the movement speed of the user 99's head.

[0092] For example, as a measure of motion of the user 99's head, detector 103 can detect rotational motion about at least one of three mutually orthogonal axes in three-dimensional space, and it can also detect displacement about at least one of the aforementioned three axes. Furthermore, as a measure of motion of the user 99's head, detector 103 can also detect both rotational and displacement.

[0093] The sensing information input unit 114 acquires the head movement speed of the user 99 from the detector 103. More specifically, the sensing information input unit 114 acquires the amount of head movement of the user 99 detected by the detector 103 per unit time as the movement speed. Thus, the sensing information input unit 114 acquires at least one of rotational speed and displacement speed from the detector 103. The amount of head movement of the user 99 acquired here is used to determine the position and orientation (in other words, coordinates and orientation) of the user 99 in the three-dimensional sound field. Therefore, the acquisition unit 111 also functions as a position acquisition unit through the sensing information input unit 114. In the sound reproduction system 100, the relative position of the sound source object with respect to the user 99 is determined based on the determined coordinates and orientation of the user 99, and sound is played. Specifically, the above functions are achieved by the path calculation unit 121 and the output sound generation unit 131.

[0094] The path calculation unit 121 includes the following functions: a direction of arrival calculation function that calculates the relative direction of arrival of the played sound from the location of the sound source object to the location of the user 99 based on the determined coordinates and orientation of the user 99; and a transformation processing function that performs the above-mentioned secondary sound calculation. Therefore, the path calculation unit 121 includes the following functions: calculating the propagation path from the sound source object, and calculating, based on the calculated propagation path of the played sound, the secondary sound that reaches the location of the user 99 through indirect propagation of the played sound and the direction of arrival of that secondary sound. Furthermore, the direction of arrival of the secondary sound includes additional information, such as, in the case of reflected sound, what kind of object reflects it, and what the attenuation rate is during reflection. This additional information is included in the direction of arrival of the secondary sound calculated from the input sound information. That is, the additional information is generated and obtained from the sound information through calculation.

[0095] The spatial information is organized and includes the spatial location of the sound source object in the space (three-dimensional sound field) (location information of the sound source object), the reflection and diffraction characteristics of the sound at the sound source object (and the condition information of the spatial environment), and also includes information such as the breadth of the three-dimensional sound field. The path calculation unit 121 generates secondary sound based on the spatial information by determining which sound source object the played sound reflects or diffracts at, and calculates the direction of arrival of the secondary sound and the volume of the secondary sound after attenuation due to reflection or diffraction as additional information. The sound information (input data) includes spatial information in the form of an audio signal and accompanying metadata. As described above, this spatial information, in addition to the audio signal, includes information necessary to convert the sound into stereo and locate the sound source object within the three-dimensional sound field, and / or information used to calculate the information necessary to convert the sound into stereo and locate the sound source object within the three-dimensional sound field.

[0096] The path calculation unit 121 can be implemented through any processing as long as it can calculate the direction of arrival of the playback sound when it reaches the user as a direct sound, and calculate the secondary sound that reaches the user's position 99 through the secondary propagation of the playback sound and its direction of arrival. Based on the coordinates and orientation of the user 99, the path calculation unit 121 determines which direction the playback sound and secondary sound should be perceived by the user 99 as coming from within the three-dimensional sound field, and processes the sound information in a way that allows the user 99 to perceive such sound when the output sound signal is reproduced.

[0097] The output sound generation unit 131 is a processing unit that generates an output sound signal by processing information related to the playback sound contained in the sound information.

[0098] Here, use Figure 5 This describes an example of the output sound generation unit 131. Figure 5 This is a block diagram illustrating the functional structure of the output sound generation unit according to an embodiment. For example... Figure 5 As shown, the output sound generation unit 131 in this embodiment includes, for example, an object determination unit 132, a first generation unit 133, a second generation unit 134, and a synthesis unit 135.

[0099] The object determination unit 132 is a processing unit that determines whether to use the first generation unit 133 or the second generation unit 134 for each of the multiple audio signals contained in the sound information. In other words, for each audio signal, the object determination unit 132 determines whether to use it for the generation of a first sound signal performed by the first generation unit 133 or for the generation of a second sound signal performed by the second generation unit 134. Therefore, the object determination unit 132 has the function of acquiring information for determining whether to use the first generation unit 133 or the second generation unit 134.

[0100] The first generation unit 133 is a processing unit used when directly convolving the played sound with a head-related transfer function without applying audio-visual processing. The first generation unit 133 is a processing unit used when generating an output sound signal in a so-called "normal mode". The first generation unit 133 acquires the played sound and a head-related transfer function corresponding to the direction of arrival of the played sound, and performs convolution processing on the played sound using the acquired head-related transfer function to generate a first sound signal.

[0101] The second generation unit 134 is a processing unit used when performing head-correlation transfer function convolution on the transformed representative sound after applying audio-visual processing to transform the played sound into a representative sound. The second generation unit 134 is also used when generating the output sound signal in a so-called "low-processing mode". The second generation unit 134 acquires the position of the played sound and the representative point, and performs the transformation processing to transform the played sound into the representative sound, so as to reproduce the played sound using the sound from the representative point.

[0102] For example, when the sound source object is located between two representative points, the sound is generated by emitting the same sound as the played sound from each of the two representative points. That is, the played sound is assigned to the two representative points. Then, by adjusting the gain of the generated sound to match the position of the sound source object, a representative tone can be generated. The transformation from the played sound to the representative tone is not limited to this example. For example, as described later, the transformation from the played sound to the representative tone can also be performed by performing time offset adjustment and gain adjustment, or any other existing transformation can be used to transform the played sound into a representative tone for reproduction by sound from the representative points. Furthermore, in this specification, the transformation process from the played sound to the representative tone can also be interpreted as the process of assigning the played sound to the representative points (representative direction). Specifically, the sound signal of the played sound associated with the position of each sound source object is assigned to the position of the representative point, and a representative tone is generated that reaches the listener from the representative point (representative direction). Here, the representative direction refers to the direction from the listener's view of the representative point's position, or the direction from the representative point's position to the listener's view of the listener. Examples of transformations involving time offset adjustment and gain adjustment will be described later. The second generation unit 134 acquires representative sounds obtained through transformation, which are the same number as the number of representative points, and head-related transfer functions corresponding to the representative directions from each representative point to the user 99's position. It then performs convolution processing on the representative sounds using the acquired head-related transfer functions to generate a second sound signal.

[0103] The synthesis unit 135 synthesizes the generated first and second audio signals to produce an output audio signal. Furthermore, the synthesis unit 135 may sometimes perform EQ adjustments on the second audio signal before synthesizing it with the first audio signal. Specifically, in sound imaging processing, EQ adjustments can be made to boost the gain of high-frequency domains considered to have a high probability of attenuation, thereby emphasizing those high-frequency domains. Therefore, the synthesis unit 135 also functions as an EQ adjustment unit. Additionally, when multiple second audio signals exist, the EQ adjustment performed in the synthesis unit 135 may be applied to only a portion of the multiple second audio signals or to all of them.

[0104] Refer again Figure 2 The output tone generation unit 131 retrieves head-related transfer functions for generating the output tone signal from the database 105. The database 105 is an information storage device that functions as both a storage device for storing information and a storage controller for reading and outputting the stored information to an external source. The database 105 stores head-related transfer functions for each direction of arrival to the user 99. The head-related transfer functions included in the database 105 can be a general set of head-related transfer functions available to everyone, a set of head-related transfer functions optimized for each of the 99 users, or a publicly released set of head-related transfer functions. The database 105 receives a request from the output tone generation unit 131 that queries the direction of arrival and outputs the head-related transfer function corresponding to that direction of arrival to the output tone generation unit 131.

[0105] The signal output unit 141 is a functional unit for outputting the generated output audio signal to the driver 104. The signal output unit 141 performs signal conversion from digital to analog based on the output audio signal, thereby generating a waveform signal. Based on the waveform signal, the driver 104 generates sound waves to provide a sound prompt to the user 99. The driver 104 includes, for example, a diaphragm and a drive mechanism such as a magnet and a voice coil. The driver 104 actuates the drive mechanism according to the waveform signal, causing the diaphragm to vibrate. In this way, the driver 104 generates sound waves (meaning "reproduces" the output audio signal, i.e., the user 99's perception is not included in the meaning of "reproduced") by the vibration of the diaphragm based on the output audio signal. The sound waves propagate through the air and reach the user 99's ear, whereupon the user 99 perceives the sound.

[0106] [Other examples of composition] In the above example, it is illustrated that the sound reproduction system 100 according to this embodiment is a sound prompting device, comprising an information processing unit 101, a communication module 102, a detector 103, a database 105, and a driver 104. However, the functions of the sound reproduction system 100 can also be implemented by multiple devices or by a single device. Specifically, the following will be used... Figures 6 to 15 Please provide an explanation. Figures 6 to 15These are diagrams used to illustrate other examples of an audio reproduction system according to an embodiment.

[0107] For example, the information processing device 601 may be included in the sound prompting device 602, which can perform sound processing and sound prompts simultaneously. Furthermore, the information processing device 601 and the sound prompting device 602 may share the responsibility of performing the sound processing described in this disclosure, or a server connected to the information processing device 601 or the sound prompting device 602 via a network may perform part or all of the sound processing described in this disclosure.

[0108] Furthermore, although referred to as information processing device 601 in the above description, when information processing device 601 decodes and performs sound processing on a bitstream generated by encoding audio signals or at least a portion of spatial information for sound processing, information processing device 601 may also be referred to as a decoding device, and sound reproduction system 100 (i.e., stereo sound reproduction system 600 in the figure) may also be referred to as a decoding processing system.

[0109] Here is an example of how the audio reproduction system 100 functions as a decoding processing system.

[0110] <Example of an encoding device> Figure 7 This is a block diagram illustrating the structure of an example of the encoding apparatus of this disclosure, namely the encoding apparatus 700.

[0111] Input data 701 is data containing spatial information and / or audio signals that is input to encoder 702 and becomes the object of encoding. Details regarding spatial information will be explained later.

[0112] Encoder 702 encodes input data 701 to generate encoded data 703. Encoded data 703 is, for example, a bitstream generated through encoding processing.

[0113] The memory 704 stores the encoded data 703. The memory 704 may be, for example, a hard disk or an SSD (Solid-State Drive), or other storage devices.

[0114] Furthermore, while the above description exemplifies the bitstream generated through encoding processing as encoded data 703 stored in memory 704, it could also be data other than bitstreams. For example, the encoding device 700 can convert the bitstream into a specified data format and store the resulting converted data in memory 704. The converted data could be, for example, a file or multiplexed stream storing one or more bitstreams. Here, a file is, for example, a file with a file format such as ISOBMFF (ISO Base Media File Format). Additionally, the encoded data 703 could also be in the form of multiple data packets generated by splitting the aforementioned bitstream or file. When converting the bitstream generated by encoder 702 into data different from the bitstream, the encoding device 700 could have a conversion unit (not shown), or the conversion processing could be performed by a CPU (Central Processing Unit).

[0115] <Example of a decoding device> Figure 8 This is a block diagram illustrating the structure of a decoding device 800, an example of the decoding device disclosed herein.

[0116] The memory 804 stores, for example, the same data as the encoded data 703 generated in the encoding device 700. The memory 804 reads the stored data and inputs it as input data 803 to the decoder 802. The input data 803 is, for example, the bitstream to be decoded. The memory 804 can be, for example, a hard disk or SSD, or other storage devices.

[0117] Alternatively, the decoding device 800 may not directly use the data stored in the memory 804 as input data 803, but instead use the transformed data generated after transforming the read data as input data 803. The data before transformation may, for example, be multiplexed data storing one or more bitstreams. Here, the multiplexed data may be a file with a file format such as ISOBMFF. Furthermore, the data before transformation may also be in the form of multiple data packets generated by splitting the aforementioned bitstream or file. When transforming data read from the memory 804 that is different from the bitstream back into a bitstream, the decoding device 800 may have a transformation unit (not shown), or the transformation processing may be performed by the CPU.

[0118] Decoder 802 decodes the input data 803 to generate an audio signal 801 that is presented to the listener.

[0119] <Other Examples of Encoding Devices> Figure 9 This is a block diagram illustrating the structure of another example of the encoding apparatus of this disclosure, namely encoding apparatus 900. Figure 9 In the middle, to and Figure 7 Structures with the same function are labeled with the same structure. Figure 7 The same reference numerals are used for the same structures, and descriptions of these structures are omitted.

[0120] The encoding device 700 has a memory 704 for storing encoded data 703, while the encoding device 900 has a transmitting unit 901 for transmitting the encoded data 703 to the outside, which is different from the encoding device 700.

[0121] The transmitting unit 901 transmits a transmission signal 902 to other devices or servers based on encoded data 703 or data in other formats generated by transforming encoded data 703. The data used to generate the transmission signal 902 may be, for example, a bit stream, multiplexed data, a file, or a data packet as described in the encoding device 700.

[0122] <Other examples of decoding devices> Figure 10 This is a block diagram illustrating the structure of another example of the decoding apparatus of this disclosure, namely the decoding apparatus 1000. Figure 10 In the middle, to and Figure 8 Structures with the same function are labeled with the same structure. Figure 8 The same reference numerals are used for the same structures, and descriptions of these structures are omitted.

[0123] The decoding device 800 has a memory 804 for reading input data 803, while the decoding device 1000 has a receiving unit 1001 for receiving input data 803 from the outside, which is different from the decoding device 800.

[0124] The receiving unit 1001 receives the receiving signal 1002 to acquire received data and outputs the input data 803 to the decoder 802. The received data may be the same as the input data 803 input to the decoder 802, or it may be data with a different data format than the input data 803. If the received data has a different data format than the input data 803, the receiving unit 1001 may convert the received data into the input data 803, or the receiving data may be converted into the input data 803 by a conversion unit (not shown) or CPU included in the decoding device 1000. The received data may be, for example, a bitstream, multiplexed data, a file, or a data packet as described in the encoding device 900.

[0125] <Decoder Function Description> Figure 11 It is shown Figure 8 or Figure 10 A block diagram of the structure of decoder 1100, an example of decoder 802 in the example.

[0126] Input data 803 is the encoded bitstream, which contains encoded audio data as the encoded audio signal and metadata for audio processing.

[0127] The spatial information management unit 1101 acquires the metadata contained in the input data 803 and parses the metadata. The metadata contains information describing the elements that act on the sound configured in the sound space. The spatial information management unit 1101 manages the spatial information required for sound processing obtained by parsing the metadata and provides the spatial information to the rendering unit 1103. In this disclosure, the information used for sound processing is referred to as spatial information, but it may also be referred to by other names. This information used for sound processing may be referred to as sound spatial information or scene information, for example. Furthermore, when the information used for sound processing changes over time, the spatial information input to the rendering unit 1103 may also be referred to as spatial state, sound spatial state, scene state, etc.

[0128] Furthermore, spatial information can be managed on a per-sound-space or per-scene basis. For example, when representing different rooms as virtual spaces, each room can be managed as a different sound-space scene, or even the same space can be managed as different scenes depending on the scene being represented. In managing spatial information, each spatial information can be assigned an identifier for identification. The spatial information data can be contained in a bitstream as a form of input data 803, or the bitstream can contain the spatial information identifier, while the spatial information data can be obtained from outside the bitstream. If the bitstream only contains the spatial information identifier, the identifier can be used during rendering to obtain the spatial information data stored in the memory of the audio signal processing device or an external server as input data.

[0129] Furthermore, the information managed by the spatial information management unit 1101 is not limited to the information contained in the bitstream. For example, input data 803 can be data not included in the bitstream, including data representing spatial characteristics or structures obtained from a software application or server providing VR or AR. Additionally, input data 803 can be data not included in the bitstream, including data representing the characteristics or location of a listener or object. Furthermore, input data 803 can include information representing the listener's location obtained by sensors in a terminal containing a decoding device, or information representing the terminal's location inferred from information obtained by sensors. In other words, the spatial information management unit 1101 can communicate with external systems or servers to obtain spatial information and the listener's location. Furthermore, the spatial information management unit 1101 can obtain clock synchronization information from external systems and perform clock synchronization processing with the rendering unit 1103. Additionally, the space described above can be a virtually formed space, i.e., VR space, or a real space or a virtual space corresponding to a real space, i.e., AR space or MR (Mixed Reality) space. Furthermore, virtual space can also be referred to as a sound field or sound space. Furthermore, the information indicating position in the above description may be information such as coordinate values ​​of the position in space, information indicating the relative position with respect to a specified reference position, or information indicating the movement or acceleration of the position in space.

[0130] The audio data decoder 1102 decodes the encoded audio data contained in the input data 803 to obtain an audio signal.

[0131] The encoded audio data acquired by the stereo sound reproduction system 600 is, for example, a bitstream encoded in a format specified by MPEG-H 3D Audio (ISO / IEC 23008-3). However, MPEG-H 3D Audio is just one example of the encoding methods that can be used when generating encoded audio data within a bitstream; bitstreams encoded using other encoding methods can also be included as encoded audio data. For example, the encoding method used can be irreversible codecs such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), and Vorbis, or reversible codecs such as ALAC (Apple Lossless Audio Codec) and FLAC (Free Lossless Audio Codec), or any encoding method other than those mentioned above. For example, PCM (Pulse Code Modulation) data can also be a type of encoded audio data. In this case, the decoding process, for example, when the quantization bit depth of the PCM data is N, is the process of converting the N-bit binary number into a digital format (e.g., floating-point format) that can be processed by the rendering unit 1103.

[0132] The rendering unit 1103 takes audio signals and spatial information as input, applies sound processing to the audio signals using the spatial information, and outputs the sound-processed audio signal 801.

[0133] Before rendering begins, the Spatial Information Management Unit 1101 reads the metadata of the input signal, detects rendering items such as objects or sounds defined by the spatial information, and sends it to the Rendering Unit 1103. After rendering begins, the Spatial Information Management Unit 1101 monitors changes in the spatial information and the listener's location over time, and updates and manages the spatial information. Then, the Spatial Information Management Unit 1101 sends the updated spatial information to the Rendering Unit 1103. Based on the audio signal contained in the input data and the spatial information received from the Spatial Information Management Unit 1101, the Rendering Unit 1103 generates and outputs an audio signal with added sound processing.

[0134] The spatial information update processing and the audio signal output processing with added sound processing can be executed in the same thread, or the spatial information management unit 1101 and the rendering unit 1103 can be assigned to separate threads. When the spatial information update processing and the audio signal output processing with added sound processing are handled by different threads, the thread start frequency can be set separately, or the processing can be executed in parallel.

[0135] By executing the spatial information management unit 1101 and the rendering unit 1103 in separate independent threads, computational resources can be prioritized for the rendering unit 1103. Therefore, even in situations where no delay is permissible, such as audio processing where even a delay of one sample (0.02 msec) would produce popping noise, it can be safely implemented. In this case, the allocation of computational resources to the spatial information management unit 1101 will be limited. However, compared to audio signal output processing, spatial information updates are low-frequency processes (e.g., updates to the listener's facial orientation). Therefore, unlike audio signal output processing, it does not require an instantaneous response, so even with limited computational resource allocation, there is little impact on the sound quality delivered to the listener.

[0136] Spatial information updates can be performed periodically at preset times or intervals, or when preset conditions are met. Furthermore, spatial information updates can be performed manually by the listener or the administrator of the sound space, or triggered by changes in the external system. For example, in cases where a listener instantly transmits their virtual avatar's standing position using the controller, instantly fast-forwards or rewinds time, or the virtual space administrator suddenly applies effects such as altering the environment, the thread configured with the spatial information management unit 1101 can be started either periodically or as a one-time interrupt handler.

[0137] The information update thread, which performs spatial information update processing, handles tasks such as updating the position or orientation of the listener's virtual avatar within the virtual space based on the position or orientation of the VR goggles worn by the listener, and updating the position of moving objects within the virtual space. These processes are completed within a processing thread that starts at a relatively low frequency of around tens of Hz. Processing reflecting the properties of direct sound can also be performed within such low-frequency processing threads. This is because the frequency of changes in the properties of direct sound is lower than the frequency of audio processing frames used for audio output. Conversely, by doing so, the computational load of the processing can be relatively reduced, and the risk of impulse noise, which would arise if information is updated at unnecessarily fast frequencies, can also be avoided.

[0138] Figure 12 It is shown Figure 8 or Figure 10Another example of decoder 802 is a block diagram of the structure of decoder 1200.

[0139] Figure 12 and Figure 11 The difference is that input data 803 contains unencoded audio signals, rather than encoded audio data. Input data 803 contains a bitstream and an audio signal, with the bitstream containing metadata.

[0140] Spatial Information Management Department 1201 and Figure 11 The same applies to the Spatial Information Management Department 1101, therefore the description is omitted.

[0141] Rendering Department 1202 and Figure 11 The rendering unit 1103 is the same, so the description is omitted.

[0142] In addition, in the above explanation, although... Figure 12 The structure is called a decoder, but it can also be called an audio processing unit that performs audio processing. Furthermore, the device containing the audio processing unit may not be called a decoding device, but rather an audio processing device. Additionally, the audio signal processing device (information processing device 601) can also be called an audio processing device.

[0143] <Physical Structure of Encoding Device> Figure 13 This is a diagram illustrating an example of the physical structure of an encoding device. Furthermore, Figure 13 The encoding device shown is an example of the aforementioned encoding devices 700 and 900.

[0144] Figure 13 The encoding device includes a processor, memory, and communication IF.

[0145] The processor may be, for example, a CPU (Central Processing Unit), a DSP (digital signal processor), or a GPU (Graphics Processing Unit), which may also implement the encoding processing of this disclosure by executing a program stored in memory. Alternatively, the processor may be a dedicated circuit for signal processing of audio signals that includes the encoding processing of this disclosure.

[0146] Memory can be composed of, for example, RAM (Random Access Memory) or ROM (Read Only Memory). Memory can also include magnetic storage media such as hard disks or semiconductor memory such as SSDs (Solid State Drives). Furthermore, internal memory built into the CPU or GPU can also be referred to as memory.

[0147] A communication interface (IF) is, for example, a communication module corresponding to communication methods such as Bluetooth (registered trademark) or WIGIG (registered trademark). The encoding device has the function of communicating with other communication devices through the communication IF and sending encoded bit streams.

[0148] The communication module, for example, consists of signal processing circuitry and an antenna corresponding to the communication method. In the above example, although Bluetooth (registered trademark) or WiGig (registered trademark) are used as examples of communication methods, it could also be a communication method corresponding to LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark). Furthermore, the communication interface may not use the wireless communication method described above, but rather a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface).

[0149] <Physical Structure of Audio Signal Processing Device> Figure 14 This is a diagram illustrating an example of the physical structure of an audio signal processing device. Additionally, Figure 14 The audio signal processing device can also be a decoding device. Furthermore, a portion of the structure described herein can also be incorporated into the sound prompting device 602. Additionally, Figure 14 The audio signal processing device shown is an example of the aforementioned audio signal processing device 601.

[0150] Figure 14 The audio signal processing device includes a processor, memory, communication IF, sensors, and speakers.

[0151] The processor may be, for example, a CPU, DSP, or GPU, which may also implement the audio processing or decoding processing of this disclosure by executing programs stored in memory. Alternatively, the processor may be a dedicated circuit for signal processing of audio signals that includes the audio processing of this disclosure.

[0152] Memory can be composed of, for example, RAM or ROM. Memory can also include magnetic storage media such as hard disks or semiconductor memory such as SSDs. In addition, internal memory built into the CPU or GPU can also be referred to as memory.

[0153] The communication IF is, for example, a communication module corresponding to communication methods such as Bluetooth (registered trademark) or WIGIG (registered trademark). Figure 14 The audio signal processing apparatus shown has the function of communicating with other communication devices via a communication IF and acquiring the bitstream of the decoded object. The acquired bitstream is stored, for example, in a memory.

[0154] The communication module may consist, for example, of a signal processing circuit and an antenna corresponding to the communication method. In the above example, although Bluetooth (registered trademark) or WiGig (registered trademark) are used as examples of communication methods, they could also correspond to LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark), etc. Furthermore, the communication interface may not use the wireless communication method described above, but rather a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark).

[0155] The sensor performs sensing to infer the listener's position or orientation. Specifically, based on the detection results of one or more of the listener's head or other part or whole body position, orientation, movement, velocity, angular velocity, or acceleration, the sensor infers the listener's position and / or orientation and generates position information representing the listener's position and / or orientation. Alternatively, the position information can represent the listener's position and / or orientation in actual space, or it can represent the displacement of the listener's position and / or orientation relative to a specified point in time. Furthermore, the position information can also represent the relative position and / or orientation with respect to the stereo sound reproduction system or an external device equipped with the sensor.

[0156] Sensors can be, for example, camera devices or ranging devices such as LiDAR (Light Detection and Ranging), or they can detect head movements by capturing images of the listener's head movements and processing those images. Furthermore, devices that use radio waves of any frequency band, such as millimeter waves, for position estimation can also be used as sensors.

[0157] in addition, Figure 14 The audio signal processing device shown can also acquire location information from an external device equipped with a sensor via communication IF. In this case, the audio signal processing device may not include a sensor. Here, an external device refers to, for example, in... Figure 6The described device includes an audio prompt device 602 or a stereoscopic image playback device worn on the listener's head. In this case, the sensor is composed of a combination of various sensors, such as a gyroscope sensor and an accelerometer sensor.

[0158] For example, as the speed of head movement of the listener, the sensor can detect the rotational angular velocity with at least one of the three mutually orthogonal axes in the sound space as the rotation axis, and it can also detect the displacement acceleration with at least one of the three axes as the displacement direction.

[0159] For example, to measure the movement of the listener's head, the sensor can detect rotation about at least one of three mutually orthogonal axes within the sound space, or displacement about at least one of these three axes. Specifically, to measure the listener's position, the sensor detects 6DoF (position (x, y, z) and angle (yaw, pitch, roll)). The sensor is a combination of various sensors used to detect motion, such as gyroscopes and accelerometers.

[0160] Alternatively, a sensor only needs to detect the listener's location; it can be implemented using a camera or a Global Positioning System (GPS) receiver. Location information obtained by inferring one's own position using LiDAR (Laser Imaging Detection and Range) or similar methods can also be used. For example, in cases where the audio signal reproduction system is implemented via a smartphone, the sensor is built into the smartphone.

[0161] In addition, sensors can also include detection capabilities. Figure 14 The temperature sensors shown include thermocouples and other temperature sensors for the audio signal processing device, as well as sensors for detecting the remaining power of batteries that are present in or connected to the audio signal processing device.

[0162] A loudspeaker, for example, has a driving mechanism such as a diaphragm, magnet, or voice coil, and an amplifier, which transmits the processed audio signal as a sound prompt to the listener. The loudspeaker uses the amplifier to amplify the audio signal (more specifically, the waveform signal representing the sound waveform) to drive the driving mechanism, which in turn causes the diaphragm to vibrate. In this way, the diaphragm vibrating according to the audio signal generates sound waves, which travel through the air and reach the listener's ear, allowing the listener to perceive the sound.

[0163] In addition, although this place is based on Figure 14The audio signal processing device shown in the illustration includes a speaker, and the example is illustrated by using this speaker to prompt the processed audio signal. However, the method of prompting the audio signal is not limited to the above structure. For example, the processed audio signal can also be output to an external sound prompting device 602 connected via a communication module. Communication via the communication module can be wired or wireless. As another example, Figure 14 The audio signal processing device shown may also have a terminal for outputting an analog signal of sound. By connecting a cable such as an earphone to the terminal, an audio signal can be received from the earphone or the like. In the above case, the audio signal is reproduced by a headphone, earbud, head-mounted display, neckband speaker, wearable speaker, or a surround speaker composed of multiple fixed speakers, etc., which are worn on the head or part of the listener's body as the sound prompting device 602.

[0164] <Functional Description of the Rendering Department> Figure 15 It is shown Figure 11 and Figure 12 A functional block diagram illustrating an example of the detailed structure of the rendering units 1103 and 1202.

[0165] The rendering unit consists of a resolution unit and a synthesis unit (different from the synthesis unit 135 mentioned above), which adds audio processing to the sound data contained in the input signal and outputs it.

[0166] The information contained in the input signal will be explained below.

[0167] The input signal may consist of spatial information, sensor information, and sound data. The input signal may also contain a bitstream of sound data and metadata (control information), in which case the metadata may include spatial information.

[0168] Spatial information is information related to the sound space (three-dimensional sound field) created by a stereo sound reproduction system. It consists of information related to objects contained in the sound space and information related to the listener. Among the objects are sound source objects that emit sound and become sound sources, and non-sound-emitting objects that do not emit sound. Non-sound-emitting objects function as obstacles that reflect the sound emitted by sound source objects, but sound source objects sometimes also function as obstacles that reflect the sound emitted by another sound source object.

[0169] The information shared by both the sound source object and the non-sound-producing object includes location information, shape information, and the volume attenuation rate when the object reflects sound.

[0170] Position information is represented by three-axis coordinates in Euclidean space, such as the X, Y, and Z axes, but it is not necessarily three-dimensional information. For example, it can also be two-dimensional information represented by two-axis coordinates, such as the X and Y axes. The position information of an object is determined by the representative position of a shape represented by a mesh or voxels.

[0171] Shape information can include information related to the surface material.

[0172] In addition, it can also include information indicating whether the object is a living organism, or whether the object is a moving object. If the object is a moving object, its position information can move over time, and the changed position information or the amount of change is transmitted to the rendering unit.

[0173] In addition to the information commonly associated with both the sound source and the non-sound-producing object mentioned above, the information related to the sound source also includes sound data and the information required to project the sound data into the sound space.

[0174] Sound data represents information related to the frequency and intensity of sound, reflecting the sound as perceived by the listener. Sound data is typically a PCM signal, but it can also be data compressed using encoding methods such as MP3. In this case, since decoding is required at least before the signal reaches the synthesis unit, the rendering unit may include a decoding unit (not shown). Alternatively, it can be decoded by the sound data decoder 1102.

[0175] For a single sound source object, only at least one sound data point needs to be set, or multiple sound data points can be set. Furthermore, identification information can be assigned to each sound data point to recognize it, and this identification information can be maintained as information related to the sound source object.

[0176] Information required for projecting sound data into the sound space may include, for example, information about a reference volume used as a reference when playing the sound data, information representing the properties (also called characteristics) of the sound data, information related to the location of the sound source object, information related to the orientation of the sound source object, and information related to the directivity of the sound emitted by the sound source object. Reference volume information, for example, is the effective value of the amplitude of the sound data at the sound source location when projecting the sound data into the sound space, and can also be expressed in decibels (dB) as a floating-point value.

[0177] For example, when the reference volume is 0dB, it can be represented as the volume of the signal level shown in the sound data remaining unchanged, and sound is emitted into the sound space from the position indicated by the aforementioned location-related information at its original volume. When it is -6dB, it can be represented as the volume of the signal level shown in the sound data being halved, and sound being emitted into the sound space from the position indicated by the aforementioned location-related information. This information is uniformly assigned to one or more sound data.

[0178] Information indicating the nature of sound data can be, for example, information related to the volume of the sound source, or information indicating its temporal variation. For instance, in a virtual conference room where the sound space is a speaker, the volume will change intermittently over short periods of time. More simply, it can be described as the alternation of audible and silent portions.

[0179] Furthermore, in a sound space like a concert hall, where the sound source is a performer, the volume will be maintained for a certain duration. Conversely, in a sound space like a battlefield, where the sound source is an explosive device, the volume of the explosion only increases momentarily before remaining silent. Thus, the information about the sound source's volume includes not only the magnitude of the sound but also the changes in magnitude, and this information can be used to represent the properties of the sound data.

[0180] Here, information about changes in sound volume can be data representing frequency characteristics in a time sequence. It can also be data representing the duration of a sound interval. Alternatively, it can be time-series data representing the duration of both the sound and silent intervals. It can also be data listing the duration during which the amplitude of the audio signal can be considered stable (or approximately constant), along with the amplitude values ​​of the signal between these periods, arranged in a time sequence. It can also be data representing the duration during which the frequency characteristics of the audio signal can be considered stable. Alternatively, it can be data listing the duration during which the frequency characteristics of the audio signal can be considered stable, along with the frequency characteristics between these periods, arranged in a time sequence.

[0181] As a data format, it could be data representing the approximate shape of a spectrum. Furthermore, the volume used as a reference for the aforementioned frequency characteristics could be set as the aforementioned reference volume. The reference volume information and the information representing the properties of the sound data, in addition to being used to calculate the volume of direct or reflected sound perceptible to the listener, can also be used for selection processing to choose whether or not to allow the listener to perceive it. Other examples of information representing the properties of the sound data and their usage in specific selection processing will be explained later.

[0182] Orientation-related information is typically represented by yaw, pitch, and roll. Alternatively, the roll rotation can be omitted, and yaw and pitch can be used instead. Orientation information can change over time and is transmitted to the rendering unit as it changes.

[0183] Information relevant to the listener relates to their position and orientation within the sound space. Position information is represented by the XYZ axes in Euclidean space, but it is not necessarily three-dimensional; it can also be two-dimensional. Orientation-related information is typically represented by yaw, pitch, and roll. Alternatively, the roll rotation can be omitted, and yaw and pitch angles can be used instead. Position and orientation information can change over time and are transmitted to the rendering unit as they change.

[0184] Sensor information includes information such as rotation or displacement detected by sensors worn by the listener, as well as the listener's position and orientation. This sensor information is transmitted to the rendering unit, which updates the listener's position and orientation based on the sensor information. Sensor information can also be location information obtained by a portable terminal through GPS, a camera, or LiDAR. Furthermore, information acquired externally via a communication module can also be used as sensor information. Information such as the temperature of the audio signal processing device and the remaining battery power can be obtained from the sensors. The computing resources (CPU capacity, memory resources, PC performance) of the audio signal processing device or audio signal prompting device can also be obtained in real time.

[0185] The analysis unit performs the same function as the acquisition unit 111 in the example above. That is, it analyzes the input signal and acquires the required information in the path calculation unit 121 and the output tone generation unit 131.

[0186] The synthesis unit performs the same functions as the path calculation unit 121, output sound generation unit 131, and signal output unit 141 in the above example. Based on the audio signal of the direct sound, and information about the arrival time and volume of the direct sound calculated by the analysis unit, the input audio signal is processed to generate the direct sound. Furthermore, based on information about the arrival time and volume of the reflected sound calculated by the analysis unit, the input audio signal is processed to generate the reflected sound. The synthesis unit synthesizes the generated direct sound and reflected sound and outputs it.

[0187] [action] Next, refer to Figure 16 The operation of the sound reproduction system 100 described above, and in particular the information processing device 101, will be explained. Figure 16This is a flowchart illustrating an example of the operation of the information processing apparatus according to the embodiment. In the example shown in the figure, the acquisition unit 111 acquires sound information through the communication module 102 (step S101). The sound information is decoded by the decoding processing unit 113 into information related to the playback sound and information related to the location of the sound source object, and an output sound signal is generated. First, the path calculation unit 121 uses the decoded sound information, specifically transforming the aforementioned sound-related information into each of a plurality of sound signals, and transforming it into sound information containing these plurality of sound signals (step S102). Each of the plurality of sound signals corresponds, for example, to a direct sound and one or more secondary sounds. In addition, if sound information containing multiple sound signals without transformation is acquired, this transformation process can be skipped. The sensing information input unit 114 acquires information related to the user's location (location information) (step S103).

[0188] Next, the object determination unit 132 selects one sound signal from multiple sound signals (step S104). Since all sound signals are selected sequentially, the selection order is not particularly limited. The object determination unit 132 determines whether the selected sound signal meets the prescribed conditions (step S105). If the prescribed conditions are met ("Yes" in step S105), the sound signal is provided to the second generation unit 134 for audio-visual processing, and a second sound signal is generated for the user 99 to perceive through a representative tone from a representative point (step S106). That is, in step S106, the second sound signal is generated by transforming the sound signal into a representative tone and convolving it with a head-related transfer function corresponding to the representative direction.

[0189] If the conditions are not met ("No" in step S105), the sound signal is provided to the first generation unit 133, where no audio-visual processing is performed. Instead, a first sound signal is generated to be perceived by the user as sound emanating from the location of the original sound source object (step S107). That is, in step S107, the first sound signal is generated by convolving the playback sound emitted from the sound source object with a head-related transfer function corresponding to the direction of arrival.

[0190] Here, the prescribed conditions are explained. These conditions, as described above, are used to distinguish whether or not audio-visual processing is performed. In other words, by using these prescribed conditions to distinguish between sound signals suitable for audio-visual processing and those unsuitable for audio-visual processing, appropriate audio-visual processing, i.e., transformation processing, can be performed. Sound signals suitable for audio-visual processing refer to sound signals that are unlikely to cause a user's sense of disharmony through audio-visual processing. Conversely, sound signals unsuitable for audio-visual processing refer to sound signals that are likely to cause a user's sense of disharmony through audio-visual processing.

[0191] For example, in the case of direct sound and secondary sound generated through the transformation described above, direct sound is more susceptible to sound degradation, while secondary sound is less susceptible to sound degradation compared to direct sound. Therefore, it is sufficient to set a condition that the sound signal associated with direct sound is not imaged, while the sound signal associated with secondary sound is imaged. The condition in this case is "(1) Belongs to secondary sound". When determining whether it belongs to secondary sound, for example, information related to the generation system can be used. Specifically, it is sufficient to determine that sound information of a generation or more than the "parent" belongs to secondary sound.

[0192] Furthermore, in secondary sound, there are secondary sounds that are easily affected by sound degradation through image processing and those that are not easily affected. For example, for sounds that emphasize the high-frequency domain, such as reverberation, image processing has the characteristic that the high-frequency domain is easily attenuated, and therefore they are easily affected by sound degradation. In this way, in secondary sound, for sound signals related to sound quality degradation, such as reverberation, which are easily affected by image processing, it is sufficient to set a specified condition to prevent image processing. The specified condition in this case is "(2) The characteristic related to sound quality degradation is not easily affected by image processing". When determining whether the characteristic related to sound quality degradation is easily affected by image processing or not easily affected, for example, it is sufficient to assign a flag for the characteristic related to sound quality degradation to each sound signal, and when the flag is ON (it is a characteristic that is easily affected by image processing), it is determined that the characteristic related to sound quality degradation is a characteristic that is easily affected by image processing. Furthermore, the aforementioned flag can be assigned during the transformation process of calculating secondary sound, or when the sound information originally contains multiple sound signals, it can be assigned when generating the sound information (on the side that sends the sound information to the communication module 102).

[0193] Furthermore, the specified conditions can be set based on the secondary sound generation system. Specifically, it can be set to inherit information about whether the next generation in the secondary sound system undergoes image processing. That is, if image processing is performed on the sound signal related to the previous generation of secondary sound, then image processing will also be performed on the sound signal related to the current generation of secondary sound and the sound signal related to the next generation of secondary sound. Conversely, if image processing is not performed on the sound signal related to the previous generation of secondary sound, then image processing will not be performed on the sound signal related to the current generation of secondary sound and the sound signal related to the next generation of secondary sound. The specified condition in this case is "(3) The sound signal related to the previous generation of secondary sound has been determined as the object of image processing". Alternatively, the specified condition "(3') The sound signal related to the previous generation of secondary sound has not been determined as the object of image processing" can be used instead of the specified condition "(3) The sound signal related to the previous generation of secondary sound has been determined as the object of image processing". Thus, if the previous generation does not undergo image processing, then the current generation will undergo image processing.

[0194] Furthermore, due to limitations in computing resources, sometimes it is necessary to perform audio-visual processing on direct sound, not only for categories such as direct sound and secondary sound. In this case, for example, a specified condition can be set based on the relative direction of the sound source object position with respect to the front of the user 99, to distinguish the sound signals related to the sound played in front of the user 99 (e.g., -45 degrees to 45 degrees with the front as 0 degrees), the sound signals related to the sound played to the side of the user 99 (e.g., -135 degrees to -45 degrees, 45 degrees to 135 degrees with the front as 0 degrees), and the sound signals related to the sound played behind the user 99 (e.g., -180 degrees to -135 degrees, 135 degrees to 180 degrees with the front as 0 degrees). The specified condition in this case is "(4) The direction of the corresponding sound source object relative to the front of the user 99 is outside the threshold range".

[0195] For example, regarding the aforementioned angle range, when it is desirable to suppress frontal sound degradation (emphasizing the sound of frontal sound sources), if the direction of the sound source associated with the sound signal is within the range of -45 degrees to 45 degrees, then no imaging processing is performed within the threshold range. On the other hand, if the direction of the sound source associated with the sound signal is within the range of -135 degrees to -45 degrees or 45 degrees to 135 degrees, then imaging processing is performed outside the threshold range. Furthermore, for example, when it is desirable to suppress sound degradation from affecting the sides (emphasizing the sound of side sound sources), if the direction of the sound source associated with the sound signal is within the range of -135 degrees to -45 degrees or 45 degrees to 135 degrees, then no imaging processing is performed within the threshold range. On the other hand, if the direction of the sound source associated with the sound signal is within the range of -180 degrees to -135 degrees or 135 degrees to 180 degrees, then imaging processing is performed outside the threshold range. Alternatively, it is also possible to configure the system to simultaneously emphasize the sound of both frontal and side sound sources.

[0196] Furthermore, for sounds emitted from a sound source that is relatively far from the user 99, or secondary sounds generated in relation to relatively distant objects, or sounds emitted from a sound source that the user 99 cannot approach, or secondary sounds generated in relation to inaccessible objects, a certain degree of sound degradation may sometimes be tolerated. Therefore, a specified condition can be set such that when the distance between the sound source or related object and the user 99 is above a threshold, the sound signal related to the sound or secondary sound is imaged. The specified condition in this case is "(5) The distance between the sound source or related object is above the threshold". In addition, the setting of this threshold is affected by external factors such as the user 99's hearing ability, so it is best to set it for each user 99 based on experience or experimentation. Alternatively, instead of this distance, the sound pressure level, gain, or level of the sound can be used, and a specified condition can be set such that when the sound pressure level, gain, or level of the sound is below the threshold compared to the threshold, imaged sound is imaged. The specified condition in this case is "(6) The sound pressure level, gain, or level of the sound is below the threshold".

[0197] The conditions specified in (1) to (6) above can be combined in more than one way as long as they do not contradict each other. For example, if the specified condition (1) or the specified condition (2) is met, the result in step S105 is "yes", or if the specified conditions (1) and the specified condition (2) are met, the result in step S105 is "yes", etc.

[0198] return Figure 16The object determination unit 132 determines whether all sound signals have been selected (step S108). If not all sound signals have been selected ("No" in step S108), the process returns to step S104, a new sound signal is selected, and the same process is repeated. If all sound signals have been selected ("Yes" in step S108), the first and second sound signals are combined (added) in the synthesis unit 135 (step S109) to generate an output sound signal. Then, the output sound signal is output by the signal output unit 141 (step S110).

[0199] [Specific examples of audio-visual processing] In the audio-visual processing disclosed herein, playback sounds from multiple sound source objects are represented by representative sounds from multiple representative directions. These representative directions can, for example, be 2 to 3 directions. Specifically, in audio-visual processing, fewer representative points than the number of sound source objects can be aggregated, and the user can perceive the playback sound as coming from the direction of arrival simply by using head-related transfer functions for the representative directions of these representative points.

[0200] At this point, in the audio-visual processing, the time offset (delay, time delay) that maximizes the cross-correlation between the head correlation transfer function from the direction of arrival of the sound source object and the head correlation transfer function representing the direction is calculated. The time offset obtained here, or the time offset with a negative sign, is assigned to the playback sound of the sound source object, and the signal after the time offset is regarded as being located in the representative direction, and then further processing is performed.

[0201] This time offset can also tolerate a time offset shorter than the sampling frequency (an offset in which the sampling point position is expressed as a decimal, hereinafter referred to as "decimal offset"). This decimal offset can be achieved through oversampling.

[0202] Here, in the audio-visual processing, a gain is applied to the signal representing the direction obtained by time-shifting the playback sound of the sound source object, and then the sum of these values ​​calculated at each representative point is convolved with the head correlation transfer function at each representative point, thereby synthesizing a signal equivalent to convolving the playback sound of the sound source object with the head correlation transfer function of the direction of arrival.

[0203] On the other hand, in audio-visual processing, when synthesizing the head-related transfer function (vector) for the direction of arrival by summing the head-related transfer functions (vectors) representing the direction, the gain can also be calculated by making the error signal vector of the synthesized head-related transfer function (vector) and the head-related transfer function (vector) for the direction of arrival orthogonal to the head-related transfer function (vector) representing the direction. Furthermore, the head-related transfer function (vector) refers to the time waveform of the head impulse response, which is the time domain representation of the head-related transfer function, as a vector. Hereinafter, this head-related transfer function (vector) will also be simply referred to as the "head-related transfer function vector".

[0204] In image processing, this gain is corrected so that the energy balance of the head correlation transfer function from the sound source object location to the left and right ears of user 99 is maintained in the head correlation transfer function actually synthesized through image processing from head correlation transfer functions from multiple representative points. In other words, in image processing, the gain can be corrected so that the energy balance of the head correlation transfer function of the left and right ears of user 99 generated by the sound source object is maintained in the head correlation transfer function actually synthesized through image processing.

[0205] In this embodiment, during audio-visual processing, for each direction of arrival of the sound source object, the gain value multiplied by the head correlation transfer function representing the direction and the time offset value applied to the head correlation transfer function representing the direction can be calculated and stored in the head correlation transfer function table described later.

[0206] Based on this, in audio-visual processing, time offset and gain values ​​corresponding to the arrival direction of each sound source object are used to time-shift and apply gain to each sound source object, and then their sum is taken as a sum signal. In audio-visual processing, this sum signal is regarded as existing at the location of the representative point. In audio-visual processing, this sum signal can be convolved with the head-related transfer function of the representative point location to generate the signal at the user's ear.

[0207] In addition, in audio-visual processing, the gain can also be calculated by minimizing the energy or L2 norm of the error signal vector between the synthesized head impulse response (HRIR) vector and the HRIR vector in the direction of the sound source.

[0208] Furthermore, the foregoing described an example of using the head correlation transfer function itself when calculating the time offset and gain that maximize cross-correlation. Alternatively, the time offset and / or gain can also be calculated using the cross-correlation value after applying a weighted filter in the frequency domain.

[0209] In other words, when calculating the time offset and gain that maximize cross-correlation, the values ​​of a frequency-weighted filter (hereinafter also referred to as a "frequency-weighted filter") can be used.

[0210] The frequency-weighted filter preferably uses a filter that cuts off frequencies near or slightly above the frequency bands where human hearing sensitivity is high, and attenuates frequencies above that band, i.e., frequency bands where human hearing sensitivity gradually decreases. For example, a low-pass filter (LPF) with a cutoff frequency of 3000Hz to 6000Hz and an attenuation rate of approximately 6dB / oct (octave) to 12dB / oct is preferred.

[0211] Furthermore, in audio-visual processing, the adjustment amounts in time offset and gain adjustment can be determined based on the head-related transfer function contained in database 105. The determined adjustment amounts are then applied to the playback sound to perform time offset and gain adjustments, transforming it into a representative sound. The optimal values ​​for the adjustment amounts in time offset and gain adjustment used in audio-visual processing vary depending on the head-related transfer function. Therefore, when reading the head-related transfer function contained in database 105, the corresponding adjustment amounts in time offset and gain adjustment are determined. This ensures that the same adjustment amounts can be used consistently whenever the head-related transfer function is subsequently used, which is advantageous from a processing power perspective.

[0212] A head-related transfer function table is an example of table data containing head-related transfer functions stored in database 105. In this table, the head-related transfer functions are stored in association with the adjustment amounts in time offset and gain adjustments determined based on those head-related transfer functions. That is, for each head-related transfer function contained in database 105, the adjustment amounts in time offset and gain adjustments can be pre-calculated, and the head-related transfer function table can be constructed. Similarly, the table data of the head-related transfer function table, which associates each head-related transfer function with its adjustment amount, can also be stored in database 105. In this way, database 105 is an example of a storage unit. Furthermore, the calculation of the adjustment amount for each head-related transfer function can also be performed by the second generation unit 134 or the decoding processing unit 113. Alternatively, the adjustment amount can be calculated by an external device and stored in the memory of the external device. In this case, the memory of the external device is equivalent to an example of a storage unit.

[0213] In addition, the adjustment amounts in time offset adjustment and gain adjustment can be pre-calculated, an adjustment amount table associated with each of the multiple representative directions can be constructed, and the table can be stored in database 105. The adjustment amount table can contain table data that associates the head-related transfer functions of the multiple representative directions with the adjustment amounts in time offset adjustment and gain adjustment, and the head-related transfer functions of the multiple representative directions can also be extracted from the head-related transfer functions of the entire celestial sphere (multiple directions) pre-acquired and stored in database 105 during rendering or system initialization.

[0214] In addition, the adjustment scale can also be a table that contains: information on which of a number of representative directions, for example, the sound signal arriving at the listener's position from the direction of each head-related transfer function in the holographic head-related transfer function database, is assigned; and information on how the audio signal should be multiplied by the time offset adjustment and gain adjustment for each representative direction during assignment.

[0215] When performing convolution processing of the head-related transfer function, by referring to the adjustment table stored in the database 105 and using the adjustment amounts in the time offset adjustment and gain adjustment associated with the head-related transfer function in the applied direction, it is not necessary to calculate the adjustment amount in each convolution processing, which helps to reduce the processing volume.

[0216] Furthermore, embodiments of the present invention can also be applied to new head-related transfer functions not included in database 105. When decoding an audio signal, turning on the power to the audio reproduction system 100, or initializing the audio reproduction system 100, the head-related transfer functions for the entire three-dimensional sound field can be newly read in, and the adjustment amount for each head-related transfer function can be calculated using the method disclosed in this embodiment or other methods. In this case, the table data associated with the head-related transfer functions and the adjustment amount can also be stored in database 105. Alternatively, the adjustment amount can be calculated by an external device and stored in the memory of the external device. When performing convolution processing of head-related transfer functions, by referring to the adjustment amounts in the time offset adjustment and gain adjustment associated with the applied head-related transfer function, it is not necessary to calculate the adjustment amount in each convolution process, thereby helping to reduce the processing load.

[0217] In this way, when reading a new head-related transfer function that is not stored in database 105, the adjustment amounts for time offset and gain adjustments used in image processing can be determined for the new head-related transfer function before it is stored in database 105. A head-related transfer function table is constructed by associating the new head-related transfer function with the determined adjustment amounts, and then this head-related transfer function table is stored in database 105. Then, during image processing, the adjustment amount is read from database 105, and offset and gain adjustments are applied based on the adjustment amount. Furthermore, the new head-related transfer function can also be a function that was previously stored in database 105 but was temporarily removed from database 105 when decoding audio signals, turning on the power to the audio reproduction system 100, or initializing the audio reproduction system 100, and then re-stored in database 105. Moreover, this determination of adjustment amounts is effective even without switching between performing image processing. In other words, a third generation unit, different from the first generation unit 133 and the second generation unit 134, can be set up instead. This third generation unit applies an adjustment amount associated with a new head-related transfer function stored in the database to perform time offset adjustment and gain adjustment on the played sound to transform it into a representative sound, and generates an output sound signal by convolving the head-related transfer function corresponding to the representative direction from each representative point position toward the user position with the representative sound.

[0218] (Other implementation methods) The implementation methods have been described above, but this disclosure is not limited to the above-described implementation methods.

[0219] For example, the sound reproduction system described in the above embodiments can be implemented as a single device possessing all the constituent elements, or it can be implemented by distributing each function to multiple devices and having these multiple devices cooperate. In the latter case, an information processing device such as a smartphone, tablet terminal, or PC can be used as the device equivalent to an information processing device. For example, in the sound reproduction system 100 which has the function of a renderer that generates audio signals with added sound effects, all or part of the renderer's functions can be performed by a server. That is, all or part of the acquisition unit 111, path calculation unit 121, output sound generation unit 131, and signal output unit 141 can exist in a server (not shown). In this case, the sound reproduction system 100 can be implemented, for example, by combining an information processing device such as a computer or smartphone, a sound prompting device such as a head-mounted display (HMD) or headphones worn by the user 99, and a server (not shown). Furthermore, the computer, sound prompting device, and server can be communicatively connected in the same network or in different networks. When connected via different networks, the possibility of communication delays increases. Therefore, processing on the server should only be permitted when the computer, sound prompting device, and server are communicably connected on the same network. Furthermore, the server can be responsible for all or part of the renderer's functions depending on the amount of bitstream data received by the sound reproduction system 100.

[0220] Furthermore, the audio reproduction system disclosed herein can also be implemented as an information processing device connected to a reproduction device equipped only with a driver, and reproduces only the output audio signal generated based on the acquired audio information to that reproduction device. In this case, the information processing device can be implemented as hardware with dedicated circuitry, or as software for causing a general-purpose processor to perform specific processing.

[0221] Furthermore, in the above embodiments, the processing performed by a specific processing unit may also be performed by other processing units. Additionally, the order of multiple processes may be changed, and multiple processes may be executed in parallel.

[0222] Furthermore, in the above embodiments, each component can also be implemented by executing a software program suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0223] Furthermore, each component can be implemented in hardware. For example, each component can be a circuit (or integrated circuit). These circuits can form a single circuit or be independent circuits. Moreover, these circuits can be general-purpose circuits or special-purpose circuits.

[0224] Furthermore, the general or specific aspects of this disclosure can also be implemented by means of apparatus, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs. Furthermore, the general or specific aspects of this disclosure can also be implemented by any combination of apparatus, methods, integrated circuits, computer programs, and recording media.

[0225] For example, this disclosure can be implemented as a sound signal reproduction method executed by a computer, or as a program for causing a computer to execute the sound signal reproduction method. This disclosure can also be implemented as a computer-readable, non-transitory recording medium containing such a program.

[0226] Furthermore, any modifications made by those skilled in the art to the various embodiments, or any combination of the constituent elements and functions of the various embodiments within the scope of this disclosure, are all included in this disclosure.

[0227] Furthermore, the encoded sound information in this disclosure can be described as a bitstream containing sound signals and metadata, wherein the sound signals are information about a specified sound reproduced by the sound reproduction system 100, and the metadata is information about the location of the sound image of the specified sound when it is positioned at a specified location within a three-dimensional sound field. For example, the sound reproduction system 100 can acquire sound information as a bitstream encoded in a specified format such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound signal contains information about a specified sound reproduced by the sound reproduction system 100. The specified sound referred to here is the sound emitted by a sound source object existing in the three-dimensional sound field or natural environmental sound, such as mechanical sounds or animal sounds including humans. Furthermore, when multiple sound source objects exist in the three-dimensional sound field, the sound reproduction system 100 will acquire multiple sound signals corresponding to the multiple sound source objects respectively.

[0228] On the other hand, metadata is, for example, information used in the sound reproduction system 100 to control the sound processing of sound signals. Metadata can also be information describing a scene represented in a virtual space (three-dimensional sound field). Here, "scene" refers to the term "collection" modeled in the sound reproduction system 100 using metadata, representing all elements of three-dimensional images and sound events in the virtual space. That is, the metadata described here can contain not only information controlling sound processing but also information controlling image processing. Of course, the metadata may contain only information controlling either sound processing or image processing, or it may contain information for controlling both. In this disclosure, such metadata is sometimes included in the bitstream acquired by the sound reproduction system 100. Alternatively, the sound reproduction system 100 may acquire metadata separately from the bitstream, as described later.

[0229] The sound reproduction system 100 uses metadata contained in the bitstream, as well as additionally acquired location information of the interactive user 99, to process the sound signal, thereby generating virtual sound effects. For example, it is conceivable to add sound effects such as initial reflection generation, subsequent reverberation generation, diffraction generation, distance attenuation effects, localization, sound image localization processing, or the Doppler effect. Furthermore, on / off information for switching all or some sound effects can also be added as metadata.

[0230] Furthermore, all or part of the metadata can also be obtained from outside the bitstream of the audio information. For example, either the metadata controlling the sound or the metadata controlling the video can be obtained from outside the bitstream, or the metadata of both can be obtained from outside the bitstream.

[0231] Furthermore, if the metadata of the control image is contained in the bitstream acquired by the audio playback system 100, the audio playback system 100 may also have the function of outputting the metadata that can be used for image control to a display device that displays an image, or a stereoscopic image playback device that reproduces a stereoscopic image.

[0232] Furthermore, as an example, the encoded metadata includes: information related to the three-dimensional sound field containing the sound source object and obstacle objects; and information related to the location of the sound image within the three-dimensional sound field when it is positioned at a specified location (i.e., perceived as sound coming from a specified direction), i.e., information related to the specified direction. Here, an obstacle object refers to an object that affects the sound perceived by the user 99 before the sound emitted by the sound source object reaches the user 99, for example, by blocking or reflecting the sound. Besides stationary objects, obstacle objects can also include animals such as people, or moving objects such as machines. Furthermore, in the case of multiple sound source objects in the three-dimensional sound field, for any given sound source object, other sound source objects may become obstacle objects. In addition, non-sound source objects such as building materials or inanimate objects, as well as sound source objects that emit sound, can all become obstacle objects.

[0233] The spatial information constituting the metadata can include not only the shape of the three-dimensional sound field, but also information representing the shape and location of obstacles and sound sources within the three-dimensional sound field. The three-dimensional sound field can be a closed or open space. The metadata includes information such as the reflectivity of structures (e.g., floors, walls, or ceilings) that can reflect sound within the three-dimensional sound field, as well as the reflectivity of obstacles within the three-dimensional sound field. Here, reflectivity is the ratio of the energy of reflected sound to the energy of incident sound, set according to the frequency band of the sound. Of course, reflectivity can also be set uniformly regardless of the frequency band of the sound. Furthermore, in the case of an open three-dimensional sound field, uniformly set parameters such as attenuation rate, diffracted sound, or initial reflection sound can also be used.

[0234] While the above description lists reflectance as a parameter related to obstacle or sound source objects within metadata, metadata can also contain information beyond reflectance. For example, metadata relating to both sound source and non-sound source objects can include information related to the object's material. Specifically, metadata can include parameters such as diffusivity, transmissivity, or absorption.

[0235] Information related to the sound source object can include volume, radiation characteristics (directivity), reproduction conditions, the number and type of sound sources emanating from the object, or information specifying the sound source region within the object. Reproduction conditions can specify, for example, whether the sound is continuously playing or triggered by an event. The sound source region within the object can be determined based on the relative position of the user 99 and the object, or it can be determined using the object as a reference. When determined based on the relative position of the user 99 and the object, the user 99 can perceive sound X emanating from the right side of the object and sound Y emanating from the left side, using the surface the user 99 is looking at as a reference. When determined using the object as a reference, the specific area of ​​the object from which a sound is emitted can be fixed, regardless of the user 99's viewing direction. For example, the user 99 can perceive a high-pitched sound from the right and a low-pitched sound from the left when looking at the object from the front. In this case, when the user 99 moves to the back of the object, they can perceive a low-pitched sound from the right and a high-pitched sound from the left when looking at the object from the back.

[0236] As spatially relevant metadata, it can include the time of initial reflection, reverberation time, or the ratio of direct sound to diffused sound. When the ratio of direct sound to diffused sound is zero, only 99% of the direct sound can be perceived by the user.

[0237] Furthermore, information indicating the position and orientation of user 99 within the 3D sound field can be included in the bitstream as initial settings and metadata, or it can be excluded from the bitstream. If the information indicating the position and orientation of user 99 is not included in the bitstream, it is obtained from information outside the bitstream. For example, if it is the position information of user 99 in VR space, it can be obtained from the application providing VR content; if it is the position information of user 99 for AR cues, it can be obtained, for example, by using a portable terminal to infer its own position via GPS, a camera, or LiDAR (Laser Imaging Detection and Ranging). Additionally, the audio signal and metadata can be stored in a single bitstream or in multiple separate bitstreams. Similarly, the audio signal and metadata can be stored in a single file or in multiple separate files.

[0238] When the audio signal and metadata are stored in multiple bitstreams, information indicating related other bitstreams can be included in one or a portion of the multiple bitstreams storing the audio signal and metadata. Furthermore, information indicating related other bitstreams can be included in the metadata or control information of each bitstream storing the audio signal and metadata. Similarly, when the audio signal and metadata are stored in multiple files, information indicating related other bitstreams or files can be included in one or a portion of the multiple files storing the audio signal and metadata. Furthermore, information indicating related other bitstreams or files can be included in the metadata or control information of each bitstream storing the audio signal and metadata.

[0239] Here, the relevant bitstreams or files are, for example, bitstreams or files that might be used simultaneously during audio processing. Furthermore, information representing other relevant bitstreams can be uniformly recorded in the metadata or control information of one bitstream among multiple bitstreams storing audio signals and metadata, or it can be separately recorded in the metadata or control information of two or more bitstreams among multiple bitstreams storing audio signals and metadata. Similarly, information representing other relevant bitstreams or files can be uniformly recorded in the metadata or control information of one file among multiple files storing audio signals and metadata, or it can be separately recorded in the metadata or control information of two or more files among multiple files storing audio signals and metadata. Alternatively, a separate control file can be generated, separately from the multiple files storing audio signals and metadata, uniformly recording information representing other relevant bitstreams or files. In this case, the control file may not store audio signals and metadata.

[0240] Here, information indicating other related bitstreams or files is included, such as identifiers representing those other bitstreams, filenames representing those other files, Uniform Resource Locators (URLs) or Uniform Resource Identifiers (URIs). In this case, the acquisition unit identifies or acquires the bitstreams or files based on the information indicating the other related bitstreams or files. Furthermore, the information indicating the other related bitstreams may be included in the metadata or control information of at least a portion of the multiple bitstreams storing audio signals and metadata; similarly, the information indicating the other related files may be included in the metadata or control information of at least a portion of the multiple files storing audio signals and metadata. Here, the file containing the information indicating the related bitstreams or files may be, for example, a control file such as a manifest file used for content distribution.

[0241] Industrial availability This disclosure is very useful in terms of user-perceived stereo sound reproduction.

[0242] Explanation of reference numerals in the attached figures 99: User 100: Sound reproduction system 101: Information processing device 102: Communication Module 103: Detector 104: Driver 105: Database 111: Acquisition Department 112: Encoded Voice Information Input Unit 113: Decoding Processing Department 114: Sensor Information Input Unit 121: Path Calculation Department 131: Output Sound Generation Unit 132: Object Determination Department 133: First Generation Department 134: Second Generation Department 135: Synthesis Department 141: Signal Output Section 300: Stereoscopic image reproduction device.

Claims

1. An information processing method, executed by a computer. The information processing method includes the following steps: The step of acquiring sound information, wherein the sound information includes audio signals and information about the location of sound source objects within a three-dimensional sound field; The step of generating the first sound signal involves using a head-related transfer function based on the location of the sound source object and the direction of arrival of the user's location within the three-dimensional sound field, along with the audio signal, to generate the first sound signal. The step of generating the second sound signal involves using the head-related transfer function based on the position of a representative point set within the three-dimensional sound field and the representative direction of the user's position, along with the audio signal, to generate the second sound signal. as well as The step of generating the output audio signal involves combining the generated first audio signal and the second audio signal to generate the output audio signal.

2. The information processing method as described in claim 1, In the step of generating the first sound signal, the first sound signal is generated by convolving a head-related transfer function corresponding to the direction of arrival with the playback sound emitted by the sound source object through the audio signal. In the step of generating the second sound signal, a transformation process is performed to transform the playback sound into a representative sound transmitted from the representative point, and the second sound signal is generated by convolving the head-related transfer function corresponding to the representative direction with the representative sound.

3. The information processing method as described in claim 2, In the transformation process, the playback sound is transformed into the representative sound by applying time offset adjustment and gain adjustment.

4. The information processing method as described in claim 1, The sound information includes multiple audio signals and the locations of multiple sound source objects that correspond one-to-one with each of the multiple audio signals. The information processing method further includes a decision-making step, which, for each of the plurality of audio signals contained in the sound information, determines whether to use it to generate the first sound signal or to use it to generate the second sound signal.

5. The information processing method as described in claim 1, The sound information includes multiple audio signals and the location of a sound source object that corresponds to each of the multiple audio signals in a many-to-one relationship. The information processing method further includes a decision-making step, which, for each of the plurality of audio signals contained in the sound information, determines whether to use it to generate the first sound signal or to use it to generate the second sound signal.

6. The information processing method as described in claim 5, The information processing method further includes a step of transforming the acquired sound information, wherein the acquired sound information is transformed into sound information comprising multiple audio signals.

7. The information processing method as described in any one of claims 4 to 6, The plurality of audio signals include audio signals associated with direct sound and audio signals associated with secondary sound, wherein the direct sound is the sound that travels directly from the location of the corresponding sound source object to the user's location, and the secondary sound is the sound that accompanies the direct sound and arrives via a different path than the direct sound. In the step of making a decision. The decision is made to use the audio signal associated with the direct sound in the generation of the first sound signal. The decision is made to use the audio signal associated with the secondary sound for the generation of the second sound signal.

8. The information processing method as described in any one of claims 4 to 6, The plurality of audio signals include audio signals associated with direct sound and audio signals associated with secondary sound, wherein the direct sound is the sound that travels directly from the location of the corresponding sound source object to the user's location, and the secondary sound is the sound that accompanies the direct sound and arrives via a different path than the direct sound. The secondary sound includes another secondary sound that is further generated accompanying the first secondary sound. In the decision-making step, for each audio signal associated with the secondary sound, a decision is made on whether to use it for the generation of the second sound signal based on information in each secondary sound related to the generation system from the direct sound.

9. The information processing method as described in any one of claims 4 to 6, In the decision-making step, for each of the plurality of audio signals, a decision is made based on the orientation of the corresponding sound source object relative to the user's frontal direction to determine whether it is used for generating the first sound signal or for generating the second sound signal.

10. The information processing method according to any one of claims 4 to 6, In the decision-making step, for each of the plurality of audio signals, a decision is made on whether to use it for generating the first audio signal or for generating the second audio signal, based on the distance from the user's location to the location of the corresponding sound source object.

11. The information processing method as described in any one of claims 4 to 6, In the decision-making step, a decision is made based on the characteristics of each of the plurality of audio signals related to the sound quality degradation when generating the second audio signal, to determine whether to use them for generating the first audio signal or for generating the second audio signal.

12. The information processing method as described in claim 1, The sound information includes the audio signal associated with the direct sound and the audio signal associated with the secondary sound. The direct sound is the sound that travels directly from the location of the corresponding sound source object to the user's location. The secondary sound is the sound that accompanies the direct sound but arrives via a different path than the direct sound. If the audio signal associated with the secondary sound is identified as reverberant based on information related to the system that generates the direct sound, it is determined not to be used for the generation of the second sound signal.

13. An information processing device comprising: The acquisition unit acquires sound information, which includes audio signals and information about the location of sound source objects within a three-dimensional sound field. The first generation unit generates a first sound signal using a head-related transfer function based on the location of the sound source object and the direction of arrival of the user's location within the three-dimensional sound field, as well as the audio signal. The second generation unit generates a second sound signal using a head-related transfer function based on the position of a representative point set in the three-dimensional sound field and the representative direction of the user's position, as well as the audio signal. as well as The synthesis unit combines the generated first and second sound signals to generate the output sound signal.

14. A program for causing a computer to perform the information processing method of claim 1.

Images