Information processing device, information processing method, and program

Abstract

It reduces the difficulty in understanding the content of the sound output by the sound source. [Solution] The information processing method acquires a stream including first position and posture information indicating the position and posture of a sound source and an audio signal indicating the sound output by the sound source (S101), acquires second position and posture information indicating the position and posture of the user's head (S102), and performs a correction to reduce the rate of change of the change rate of the position or posture indicated in the acquired second position and posture information with respect to the position or posture of the sound source indicated in the first position and posture information, thereby acquiring second position and posture information used for stereophonic sound processing of the audio signal using the first position and posture information and the second position and posture information (S103).

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is a technology that processes a sound signal to be output (also referred to as stereophonic processing) according to the position and orientation of a sound source and the position and orientation of a user who is a listener, and allows the user to experience stereophonic sound (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when the sound source position recognized by the user based on the sound signal subjected to stereophonic processing changes rapidly, there is a problem that it is difficult for the user to hear the content of the sound output from the sound source.

[0006] Therefore, the present invention provides an information processing method and the like that suppresses the difficulty of hearing the content of the sound output from the sound source.

Means for Solving the Problems

[0007] An information processing apparatus according to one aspect of the present invention is an information processing apparatus comprising a processor and a memory, wherein the processor uses the memory to acquire a stream including first position and orientation information indicating the position and orientation of a sound source and an audio signal indicating the sound output by the sound source, acquires second position and orientation information indicating the position and orientation of a user's head, generates an update of the user's head position or orientation based on the acquired second position and orientation information as a time-interpolated position or orientation, and performs stereophonic sound processing on the audio signal using the first position and orientation information and the time-interpolated second position and orientation information.

[0008] These comprehensive or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0009] The information processing method of the present invention can suppress the difficulty in hearing the content of the sound output by the sound source. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is an explanatory diagram showing an example of the positional relationship between the user and the sound source in an embodiment. [Figure 2] Figure 2 is a block diagram showing the functional configuration of the information processing device in the embodiment. [Figure 3] Figure 3 is an explanatory diagram of the spatial resolution of stereophonic sound processing in the embodiment. [Figure 4] Figure 4 is an explanatory diagram of the time response length of stereophonic sound processing in the embodiment. [Figure 5] Figure 5 is an explanatory diagram showing a first example of the parameters for stereophonic sound processing in the embodiment. [Figure 6] Figure 6 is the first explanatory diagram showing the change in yaw angle in the embodiment. [Figure 7] FIG. 7 is a second explanatory diagram showing the change in the yaw angle in the embodiment. [Figure 8] FIG. 8 is a flowchart showing the processing of the information processing apparatus in the embodiment. [Figure 9] FIG. 9 is a block diagram showing the functional configuration of the information processing apparatus in a modification of the embodiment. [Figure 10] FIG. 10 is an explanatory diagram showing the change in the yaw angle and the delay of the sound signal in a modification of the embodiment. [Figure 11] FIG. 11 is a flowchart showing the processing of the information processing apparatus in a modification of the embodiment. MODE FOR CARRYING OUT THE INVENTION

[0011] (Knowledge on which the present invention is based) The present inventor has found that the following problems occur with respect to the stereophonic sound processing described in the “Background Art” section.

[0012] In the stereophonic sound processing technology shown in Patent Document 1, future predicted posture information is acquired based on the user's posture, and media content is pre-rendered using the predicted posture information.

[0013] However, when the sound source position recognized by the user based on the sound signal subjected to stereophonic sound processing changes rapidly, there is a problem that it is difficult for the user to hear the content of the voice output from the sound source. A rapid change in the sound source position recognized by the user can occur, for example, when the user suddenly changes the posture of the head by turning the neck, or when the user suddenly changes the posture of the head by moving the upper body or lower body.

[0014] In order to solve the above problems, an information processing apparatus according to an aspect of the present invention is an information processing apparatus including a processor and a memory. The processor uses the memory to acquire a stream including first position and orientation information indicating the position and orientation of a sound source, and a sound signal indicating the sound output by the sound source, acquire second position and orientation information indicating the position and orientation of the user's head, generate an update of the position or orientation of the user's head based on the acquired second position and orientation information as a temporally interpolated position or orientation, and perform stereophonic processing on the sound signal using the first position and orientation information and the temporally interpolated second position and orientation information.

[0015] In order to solve the above problems, an information processing method according to an aspect of the present invention is an information processing method executed by an information processing apparatus including a processor and a memory. The processor uses the memory to acquire a stream including first position and orientation information indicating the position and orientation of a sound source, and a sound signal indicating the sound output by the sound source, acquire second position and orientation information indicating the position and orientation of the user's head, generate an update of the position or orientation of the user's head based on the acquired second position and orientation information as a temporally interpolated position or orientation, and perform stereophonic processing on the sound signal using the first position and orientation information and the temporally interpolated second position and orientation information.

[0016] In order to solve the above problems, an information processing method according to an aspect of the present invention is to acquire a stream including first position and orientation information indicating the position and orientation of a sound source, and a sound signal indicating the sound output by the sound source, acquire second position and orientation information indicating the position and orientation of the user's head, and perform correction to reduce the rate of change of the rate of change of the position or orientation indicated by the acquired second position and orientation information with respect to the position or orientation indicated by the first position and orientation information, thereby obtaining the second position and orientation information used for stereophonic processing on the sound signal using the first position and orientation information and the second position and orientation information.

[0017] According to the above embodiment, since stereophonic sound processing is performed using the corrected position or posture of the user's head, it is possible to suppress relatively large changes in the sound heard by the user that may occur when there is a relatively large change in the position or posture of the user's head. As a result, large changes in the position of the sound source that the user hears and recognizes are suppressed, making it easier for the user to hear the content of the sound emitted by the sound source. Thus, according to the above information processing method, it is possible to suppress the difficulty in hearing the content of the sound emitted by the sound source.

[0018] For example, in the correction, if the rate of change exceeds a threshold, the second position and attitude information may be corrected so that the rate of change of the position or attitude change rate shown in the corrected second position and attitude information becomes the threshold.

[0019] According to the above embodiment, when the rate of change in the speed of change of the user's head position or posture relative to the sound source exceeds a threshold, the position or posture information is corrected so that the rate of change becomes the threshold, thereby making it possible to keep the rate of change in the speed of change of the user's head position or posture relative to the sound source below the threshold. As a result, it is possible to suppress relatively large changes in the sound heard by the user that may occur when there is a large change in the user's head position or posture that exceeds a predetermined standard. In this way, the above information processing method can suppress the difficulty in hearing the content of the sound output by the sound source.

[0020] For example, in the correction, if the rate of change exceeds a threshold, the second position and attitude information may be corrected so that the corrected second position and attitude information indicates a position or attitude that is delayed compared to the position or attitude indicated in the acquired second position and attitude information.

[0021] According to the above embodiment, when the rate of change in the speed of change of the user's head position or posture relative to the sound source exceeds a threshold, the change is corrected to slow it down, thereby keeping the rate of change in the speed of change of the user's head position or posture relative to the sound source below the threshold. As a result, relatively large changes in the sound heard by the user, which may occur when there is a large change in the user's head position or posture exceeding a predetermined standard, can be suppressed. In this way, the above information processing method can suppress the difficulty in hearing the content of the sound output by the sound source.

[0022] For example, the rate of change of the rate of change of the position or the attitude may be the second derivative of the position or attitude with respect to time.

[0023] According to the above embodiment, the rate of change of the position or posture can be easily obtained using the second derivative of the user's head position or posture with respect to the sound source, and the user's head position or posture can be appropriately corrected using this rate of change. Therefore, according to the above information processing method, the difficulty in hearing the content of the sound output by the sound source can be more easily suppressed.

[0024] For example, the stream further includes classification information indicating whether the sound indicated by the sound signal is human speech, and the correction may be performed after changing the threshold to a smaller value if the classification information indicates that the sound indicated by the sound signal is human speech.

[0025] According to the above embodiment, in the spatial audio processing of human speech, a smaller threshold is used for correction, thereby suppressing large changes in the rate of change of the user's head position or posture relative to the sound source, especially with respect to speech. Therefore, according to the above information processing method, the difficulty in hearing the content of human speech output by the sound source can be further suppressed.

[0026] For example, the stream further includes classification information indicating whether the sound indicated by the sound signal is human speech, and in the correction, if the classification information indicates that the sound indicated by the sound signal is not human speech, the threshold may be changed to a larger value before the correction is performed.

[0027] According to the above embodiment, in spatial audio processing for non-human speech, a larger threshold is used for correction, thereby allowing for larger changes in the speed of changes in the user's head position or posture relative to the sound source, and thus reducing the delay in changes in the user's head position or posture. Compared to human speech, there is an advantage in being able to reduce the delay in spatial audio processing when there is less need to make the content of non-human speech easier to hear. Therefore, according to the above information processing method, it is possible to suppress the difficulty in hearing the content of the sound output by the sound source while suppressing the delay in spatial audio processing.

[0028] For example, the stream further includes type information indicating whether the sound indicated by the sound signal is human speech, and the correction may be prohibited if the type information indicates that the sound indicated by the sound signal is not human speech.

[0029] According to the above embodiment, since no correction is made in the spatial audio processing of non-human voices, there is no delay in changes in the user's head position or posture. Compared to human voices, there is an advantage in that the delay in spatial audio processing can be further reduced when there is less need to make the content of non-human voices easier to hear. Therefore, according to the above information processing method, it is possible to suppress the difficulty in hearing the content of the sound output by the sound source while suppressing the delay in spatial audio processing.

[0030] For example, the correction may further include a delay process that delays the sound signal by a delay time, which is the time by which the change in position or attitude indicated in the second position and attitude information is delayed by the correction.

[0031] According to the above embodiment, the sound signal is delayed by a delay time that delays the change in position or posture indicated in the second position and posture information through correction, so that the time difference that may occur between the stereophonic sound processing based on the position or posture of the user's head and the sound signal to which the stereophonic sound processing is to be applied can be suppressed. Therefore, according to the above information processing method, the difficulty in hearing the content of the sound output by the sound source can be further suppressed.

[0032] For example, in the correction, a suppression process may be applied to the subsequent signal, which is an audio signal that follows the audio signal that has undergone the delay processing, to suppress the delay caused by the delay processing.

[0033] According to the above embodiment, the delay of the sound signal caused by the delay processing is recovered by the suppression processing. Therefore, according to the above information processing method, the difficulty in hearing the content of the sound output by the sound source can be further suppressed.

[0034] Furthermore, an information processing apparatus according to one aspect of the present invention is an information processing apparatus comprising: a decoding unit that acquires a stream including first position and orientation information indicating the position and orientation of a sound source and an audio signal indicating the sound output by the sound source; an acquisition unit that acquires second position and orientation information indicating the position and orientation of the user's head; and a correction unit that acquires second position and orientation information used for stereophonic sound processing of the audio signal using the first position and orientation information by performing a correction to reduce the rate of change of the change rate of the position or orientation indicated in the acquired second position and orientation information with respect to the position or orientation of the sound source indicated in the first position and orientation information.

[0035] According to the above embodiment, the same effects as the above information processing method are achieved.

[0036] Furthermore, a program according to one aspect of the present invention is a program that causes a computer to execute the above-described information processing method.

[0037] According to the above embodiment, the same effects as the above information processing method are achieved.

[0038] These comprehensive or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, or recording medium.

[0039] The embodiments will be described in detail below with reference to the drawings.

[0040] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0041] (Embodiment) In this embodiment, we will describe an information processing method and information processing device that suppress the difficulty in hearing the content of the sound output by the sound source.

[0042] Figure 1 is an explanatory diagram showing an example of the positional relationship between user U and sound source 5 in this embodiment.

[0043] Figure 1 shows user U in space S and sound source 5 perceived by user U. In Figure 1, space S is represented as a plane containing the x and y axes, but it also extends in the z-axis direction. The same applies hereafter.

[0044] The space S may contain walls or objects. Walls include ceilings or floors.

[0045] The information processing device 10 (see Figure 2 below) generates an audio signal for user U to hear by performing 3D audio processing, which is digital audio processing, based on a stream containing the audio signal output by the sound source 5. The stream further includes position and orientation information indicating the position and orientation of the sound source 5 in space S. The audio signal generated by the information processing device 10 is output as sound by a speaker and heard by user U. The speaker is assumed to be, but is not limited to, the speaker of earphones or headphones worn by user U.

[0046] Sound source 5 is a virtual sound source (also commonly called a sound image) that user U, upon hearing the sound signal generated based on the stream, recognizes as a sound source; in other words, it is not the actual source that is producing the sound. Although a human is shown as sound source 5 in Figure 1, sound source 5 is not limited to a human and can be any sound source.

[0047] User U listens to sound output from a speaker, which is sound based on an audio signal generated by the information processing device 10.

[0048] The sound output from the speaker based on the sound signal generated by the information processing device 10 is heard by each of user U's left and right ears. The information processing device 10 provides an appropriate time difference or phase difference (also referred to as time difference, etc.) to the sound heard by each of user U's left and right ears. User U perceives the direction of the sound source 5 from user U's perspective based on the time difference, etc., of the sound heard by each of the left and right ears.

[0049] Furthermore, the information processing device 10 includes, in addition to the sound heard by each of user U's left and right ears, sounds corresponding to the sound arriving directly from sound source 5 (referred to as direct sound) and sounds corresponding to the sound emitted by sound source 5 and reflected by the wall (referred to as reflected sound). User U perceives the distance from user U to sound source 5 based on the time intervals of the direct sound and reflected sound contained in the sound heard.

[0050] In the spatial audio processing performed by the information processing device 10, based on the sound signals included in the stream, the timing at which the direct sound and reflected sound arrive at the user U, as well as the amplitude and phase of the direct sound and reflected sound are calculated, and the direct sound and reflected sound are combined to generate a sound signal (referred to as the output signal) that indicates the sound to be output from the speaker.

[0051] When the user's posture changes relatively rapidly in relation to sound source 5, user U may have difficulty hearing the content of the sound output from the speaker, or may be unable to hear it at all. Therefore, it is necessary to ensure that user U can hear the content of the sound output from the speaker.

[0052] Furthermore, sound signals may include human voices. In such cases, user U may have difficulty hearing the content of the voice output from the speaker, or may be unable to hear it at all. Generally, user U's need to hear the content of voices is higher than their need to hear other sounds. Therefore, it is also necessary to ensure that user U can hear the content of the voices output from the speaker. Here, "voice" refers to human voices.

[0053] The information processing device 10 contributes to suppressing the difficulty in hearing the content of the sound output by the sound source by adjusting the position or posture based on the rate of change of the relative position or posture change speed between the user U and the sound source 5.

[0054] Figure 2 is a block diagram showing the functional configuration of the information processing device 10 in this embodiment.

[0055] As shown in Figure 2, the information processing device 10 includes, as functional units, a decoding unit 11, an acquisition unit 12, an adjustment unit 13, a processing unit 14, and a correction unit 15. The functional units of the information processing device 10 can be realized by the processor (CPU (Central Processing Unit), etc.) (not shown) of the information processing device 10 executing a predetermined program using memory (not shown).

[0056] The decoding unit 11 is a functional unit that decodes the stream. Specifically, the stream includes position and orientation information (corresponding to first position and orientation information) indicating the position and orientation of the sound source 5 in space S, and an audio signal indicating the sound output by the sound source 5. The stream may also include type information indicating whether or not the sound output by the sound source 5 is human speech.

[0057] The decoding unit 11 provides the audio signal obtained by decoding the stream to the processing unit 14, and also provides the position and orientation information obtained by decoding the stream to the adjustment unit 13. The stream may be acquired by the information processing device 10 from an external device, or it may be stored in advance in the memory device of the information processing device 10.

[0058] A stream is a stream encoded in a predetermined format, such as a stream encoded in the MPEG-H 3D Audio (ISO / IEC 23008-3) format (also simply referred to as "MPEG-H 3D Audio").

[0059] The position and orientation information indicating the position and orientation of sound source 5 more specifically includes the coordinates of sound source 5 in the three axes (x, y, and z) and angles around the three axes (yaw angle, pitch angle, and roll angle), resulting in information of six degrees of freedom. The position and orientation information of sound source 5 makes it possible to determine the position and orientation of sound source 5. Note that the coordinates are coordinates in a coordinate system that is set appropriately. The orientation is the angle around the three axes that indicates a predetermined direction (referred to as the reference direction) for sound source 5. The reference direction may be the direction in which sound source 5 emits sound, or any other direction that can be uniquely determined for sound source 5.

[0060] The stream may include, for each of the one or more sound sources 5, position and orientation information indicating the position and orientation of the sound source 5, and an audio signal indicating the sound output by the sound source 5.

[0061] The acquisition unit 12 is a functional unit that acquires the position and orientation of the user U's head in space S. The acquisition unit 12 acquires position and orientation information (second position and orientation information), which includes information indicating the position of the user U's head (referred to as position information) and information indicating its orientation (referred to as orientation information), using sensors or the like. The position and orientation information of the user U's head acquired by the acquisition unit 12 may be corrected by the correction unit 15 (described later). The acquisition unit 12 provides the position and orientation information of the user U's head to the adjustment unit 13. The position and orientation information that the acquisition unit 12 provides to the adjustment unit 13 is the acquired position and orientation information of the user U's head, and if correction has been made by the correction unit 15, it is the corrected position and orientation information of the user U's head.

[0062] The position and orientation information of user U's head more specifically includes 6 degrees of freedom information, which includes the coordinates of user U's head in the three axes (x, y, and z) and angles around the three axes (yaw angle, pitch angle, and roll angle). The position and orientation information of user U's head allows for the determination of user U's head position and orientation. The coordinates are in a coordinate system common to the coordinate system defined for sound source 5. Position can be defined as a position relative to a predetermined position in the coordinate system (e.g., the origin). Orientation is the angle around the three axes indicating the direction user U's head is facing.

[0063] The sensors may include, for example, an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetic sensor, or a combination thereof. The sensors are intended to be attached to the user U's head and may be fixed to earphones or headphones worn by the user U.

[0064] The adjustment unit 13 is a functional unit that adjusts the position and orientation information of user U in space S using parameters (i.e., spatial resolution and time response length) in the stereophonic sound processing performed by the processing unit 14. The adjustment unit 13 adjusts the position information of user U's head acquired by the acquisition unit 12 by changing it to one of the integer multiples of the spatial resolution. When making the change, the adjustment unit 13 may adopt the value closest to the position information of user U's head acquired by the acquisition unit 12 from among several integer multiples of the spatial resolution. The adjustment unit 13 provides the adjusted position information of user U's head and the orientation information of user U's head to the processing unit 14.

[0065] The processing unit 14 is a functional unit that applies stereophonic sound processing, which is digital sound processing, to the sound signal acquired by the decoding unit 11. The processing unit 14 has multiple filters used for stereophonic sound processing. The filters are used, for example, to perform calculations that adjust the amplitude and phase of the sound signal for each frequency.

[0066] The processing unit 14 calculates the propagation paths of direct and reflected sound arriving from the sound source 5 to user U in spatial audio processing, and also calculates the timing of when the direct and reflected sound arrive at the user. Furthermore, for each angular range centered on user U's head, the processing unit 14 applies a filter appropriate to the range to the signal indicating the sound (direct and reflected sound) arriving at user U from that range, thereby calculating the amplitude and phase of the sound arriving at user U.

[0067] The processing unit 14 performs stereophonic sound processing using the relative position and orientation of the user U and the sound source 5. The relative position and orientation of the user U and the sound source 5 are represented by vectors indicating the position and orientation of the sound source 5.

number

number

number

[0068] The correction unit 15 corrects the information indicating the position and orientation of the user U's head, which has been acquired by the acquisition unit 12. Specifically, the correction unit 15 corrects the rate of change of the position or orientation change rate indicated in the information indicating the position and orientation of the user U's head (corresponding to the second position and orientation information) provided by the acquisition unit 12 to reduce the rate of change of the position or orientation change rate indicated in the corrected second position and orientation information. Specifically, the correction performed by the correction unit 15 may be such that, if the rate of change exceeds a threshold, the rate of change of the position or orientation change rate indicated in the corrected second position and orientation information becomes the threshold. The correction performed by the correction unit 15 can also be said to be a correction that suppresses rapid changes in the position or orientation indicated in the second position and orientation information. Here, the threshold may be determined in accordance with a predetermined standard relating to the rate of change of the position or orientation change rate.

[0069] Furthermore, the correction performed by the correction unit 15 may be a correction that, when the rate of change exceeds a threshold, causes the corrected second position and attitude information to show a position or attitude that is delayed compared to the position or attitude shown in the acquired second position and attitude information. Here, the rate of change of the rate of change of position or attitude can be calculated, for example, as the second derivative of the position or attitude with respect to time.

[0070] Furthermore, the correction unit 15 may change the threshold to a smaller value before correction if the type information indicates that the sound indicated by the sound signal is a human voice. Also, the correction unit 15 may change the threshold to a larger value before correction if the type information indicates that the sound indicated by the sound signal is not a human voice.

[0071] Furthermore, the correction unit 15 may choose not to perform correction, or in other words, may prohibit correction, if the type information indicates that the sound signal is not a human voice.

[0072] The spatial resolution of stereophonic sound processing will be explained with reference to Figure 3.

[0073] Figure 3 is an explanatory diagram of the spatial resolution and temporal response length of the stereophonic sound processing in this embodiment.

[0074] As shown in Figure 3, the spatial resolution of stereophonic audio processing is the resolution in the angular range centered on the user U.

[0075] The processing unit 14 calculates sound signals representing the sound arriving at user U from each of the angular ranges 30, 31, 32, ... by applying filters corresponding to each angular range to the sound signal, centered on user U (see Figure 3). The sound arriving at user U from each of the angular ranges 30, 31, 32, ... may include direct sound and reflected sound from sound source 5 to user U.

[0076] Here, high spatial resolution corresponds to a narrow angular range, and conversely, low spatial resolution corresponds to a wide angular range. The angular range corresponds to the unit to which the same filter is applied.

[0077] The time response length of stereophonic sound processing will be explained with reference to Figure 4.

[0078] Figure 4 is an explanatory diagram of the response time length of the stereophonic sound processing in this embodiment.

[0079] Figure 4 shows the sound signal generated by 3D audio processing. The sound signal includes waveform 51, which corresponds to the direct sound arriving from sound source 5 to user U, and waveforms 52, 53, 54, 55, and 56, which correspond to the reflected sound arriving from sound source 5 to user U. Each of the waveforms 52, 53, 54, 55, and 56 corresponding to the reflected sound is delayed from the direct sound by a delay time determined by the positional relationship between sound source 5, user U, and the wall surface in space S, and its amplitude is reduced due to the distance of propagation and reflection from the wall surface. The delay time is set in the range of approximately 10 msec to 100 msec.

[0080] Time response length is an indicator of the magnitude of the delay time. A longer time response length results in a larger delay time, while a shorter time response length results in a smaller delay time.

[0081] It should be noted that the time response length is merely an indicator of the magnitude of the delay time and does not represent the delay time of the waveform corresponding to the reflected sound itself. For example, in Figure 4, the time width from waveform 51 to waveform 55 and the time response length are approximately equal, but this is not limited to this case. The time width from waveform 51 to waveform 54 and the time response length may be approximately equal, or the time width from waveform 51 to waveform 56 and the time response length may be approximately equal.

[0082] Figure 5 is an explanatory diagram showing the parameters for stereophonic sound processing in this embodiment.

[0083] Figure 5 shows a correspondence table in which spatial resolution and temporal response length, which are parameters of stereophonic sound processing, are associated with each of several ranges of the distance D between user U and sound source 5.

[0084] In Figure 5, a larger distance D between user U's head and sound source 5 corresponds to a lower spatial resolution. Furthermore, a larger distance D between user U's head and sound source 5 corresponds to a longer time response length.

[0085] For example, a distance D of less than 1m is associated with a spatial resolution of 10 degrees and a time response length of 10 msec.

[0086] Similarly, distances D of 1m to less than 3m, 3m to less than 20m, and 20m or more are associated with spatial resolutions of 30°, 45°, and 90°, and time response lengths of 50msec, 200msec, and 1sec, respectively.

[0087] The processing unit 14 holds a correspondence table between distance D and spatial resolution shown in Figure 5, and refers to this table to obtain the spatial resolution and time response length associated with the distance D between the user U's head and the sound source 5, which was obtained from the acquisition unit 12.

[0088] In this way, the processing unit 14 sets the spatial resolution lower the greater the distance D between the user U's head and the sound source 5 in space S, in other words, sets a value that indicates a lower spatial resolution. Also, the processing unit 14 sets the time response length higher the greater the distance D between the user U's head and the sound source 5 in space S, in other words, sets a value that indicates a longer time response length.

[0089] The correction of position and attitude information by the correction unit 15 will be explained below. Here, the yaw angle, which is the angle around the z-axis of the user U's head, will be used as the position information, but the same explanation will also apply to the coordinates of the user U's head (x, y or z), or other angles (pitch angle or roll angle).

[0090] Figure 6 is the first explanatory diagram showing the change in yaw angle in this embodiment. Figure 6 shows the temporal change in the yaw angle 60 of the user U's head, acquired by the acquisition unit 12. The yaw angle 60 shown in Figure 6 represents the relative position of the user U's head with respect to the position of the sound source 5.

[0091] As shown in Figure 6, the yaw angle of 60 is constant at ψ1 before time T1, increases linearly with time to ψ2 between time T1 and time T2, and remains constant at ψ2 from time T2 onward. Here, the slope of ψ(t) changes discontinuously at times T1 and T3. In other words, the attitude changes rapidly at times T1 and T3, or to put it another way, the rate of change of the attitude change is large.

[0092] Figure 7 is a second explanatory diagram showing the change in yaw angle in this embodiment. Figure 7 shows the temporal changes in yaw angles 61 and 62 after the correction unit 15 corrects the yaw angle 60 shown in Figure 6.

[0093] The yaw angle 61 is obtained as a result of the correction unit 15 correcting the yaw angle 60 using a relatively large threshold. The yaw angle 62 is obtained as a result of the correction unit 15 correcting the yaw angle 60 using a relatively small threshold. The "relatively small threshold" is smaller than the "relatively large threshold".

[0094] The correction unit 15, for example, applies a relatively small threshold to human speech and a relatively large threshold to non-human speech. The correction unit 15 refers to type information about the sound signal to be corrected and changes the threshold to a smaller value if it determines that the sound signal to be corrected is human speech, while changing the threshold to a larger value if it determines that the sound signal to be corrected is not human speech.

[0095] The yaw angle 61 is constant at ψ1 before time T1, increases gradually between time T1 and time T2, and is constant at ψ2 from time T3 onward.

[0096] This temporal change in yaw angle 61 is obtained by the correction unit 15 applying a correction to the temporal change in yaw angle 60 acquired by the acquisition unit 12 to suppress rapid changes in attitude.

[0097] More specifically, the yaw angle 61 is obtained by correcting the rate of change ψ''(t) of the rate of change ψ'(t) of the yaw angle ψ(t) with respect to time, which is obtained from the yaw angle ψ(t) repeatedly acquired by the acquisition unit 12, so that it is below a threshold.

[0098] For example, using the temporal change ψ(t) of the yaw angle 60 acquired by the acquisition unit 12, the rate of change of the yaw angle ψ(t) with respect to time ψ'(t) is: ψ'(t)=ψ(t) / Δt It is expressed as follows, and the rate of change ψ''(t) with respect to time of the rate of change ψ'(t) is, ψ''(t)=ψ'(t) / Δt This is expressed as follows. Here, Δt is the time difference between the time when the yaw angle ψ(t-1) was obtained last time and the time when the yaw angle ψ(t) was obtained this time, and is, for example, about 10 msec to 100 msec.

[0099] If Δt can be considered sufficiently small with respect to the change in the user U's head posture, the rate of change ψ''(t) can be calculated as the second derivative of the yaw angle ψ(t) with respect to time.

[0100] When the acquisition unit 12 acquires the temporal change ψ(t) of the yaw angle 60, the correction unit 15 calculates ψ'(t) and then calculates ψ''(t). The correction unit 15 then determines whether ψ''(t) exceeds the threshold Th1, and if it determines that it exceeds the threshold Th1, it corrects by calculating a yaw angle that makes ψ''(t) less than or equal to the threshold Th1 and setting that as ψ(t). More specifically, the correction unit 15 corrects by calculating a yaw angle that makes ψ''(t) equal to the threshold Th1 and setting that as ψ(t).

[0101] Furthermore, if ψ(t) has been corrected, the correction unit 15 uses the corrected ψ(t) to determine whether correction is necessary for the yaw angle ψ(t+1) to be acquired next, in the same manner as described above, and performs the correction if necessary.

[0102] In this way, the temporal change of the yaw angle 61 shown in Figure 7 is obtained. In the temporal change of the yaw angle 61, the discontinuity in the slope of ψ(t) at times T1 and T3 that was included in the temporal change of the yaw angle 60 is resolved; in other words, the slope of the temporal change of the yaw angle 61 changes gradually.

[0103] Next, the yaw angle of 62 is constant at ψ1 before time T1, increases gradually between time T1 and time T2, and is constant at ψ2 from time T4 onward. Time T4 is a time that has progressed from time T3.

[0104] This temporal change in yaw angle 62 is obtained by the correction unit 15 applying a correction to the temporal change in yaw angle 60 acquired by the acquisition unit 12, thereby suppressing abrupt attitude changes. The intensity of the correction applied by the correction unit 15 when obtaining the temporal change in yaw angle 62 is greater than the intensity of the correction applied by the correction unit 15 when obtaining the temporal change in yaw angle 61. In other words, the threshold Th2 used by the correction unit 15 when obtaining the temporal change in yaw angle 62 is smaller than the threshold Th1 used by the correction unit 15 when obtaining the temporal change in yaw angle 61.

[0105] As a result, in the temporal change of yaw angle 62, the discontinuity in the slope of ψ(t) at times T1 and T3 that was present in the temporal change of yaw angle 60 is eliminated; in other words, the slope of the temporal change of yaw angle 62 changes even more gradually.

[0106] The calculation process by which the correction unit 15 obtains the temporal change of the yaw angle 62 is equivalent to using threshold Th2 instead of threshold Th1 in the calculation process for obtaining the temporal change of the yaw angle 62, so a detailed explanation is omitted.

[0107] Figure 8 is a flowchart showing the processing of the information processing device 10 in this embodiment.

[0108] As shown in Figure 8, in step S101, the decoding unit 11 acquires a stream. The stream includes information indicating the position and orientation of the sound source 5 (corresponding to the first position and orientation information) and an audio signal indicating the sound output by the sound source 5.

[0109] In step S102, the acquisition unit 12 acquires information indicating the position and orientation of the user U's head (corresponding to second position and orientation information).

[0110] In step S103, the correction unit 15 corrects the information indicating the position and posture of the user U's head, which was acquired by the acquisition unit 12 in step S102. The correction is to make the rate of change of position or posture shown in the above information below a threshold.

[0111] In step S104, the processing unit 14 uses the position or orientation corrected in step S103 to perform stereophonic processing on the sound signal, thereby generating and outputting the sound signal that the speaker should output. The output sound signal is then transmitted to the speaker, output as sound, and is expected to be heard by user U.

[0112] As a result, the information processing device 10 can suppress the difficulty in hearing the content of the sound output by the sound source.

[0113] (Modified example of the embodiment) In this modified example, we describe an information processing device that suppresses the difficulty in hearing the content of the sound output by the sound source, and further describes a form that suppresses the time difference in the timing of the sound signal to which spatial audio processing is applied.

[0114] Figure 9 is a block diagram showing the functional configuration of the information processing device 10A in this modified example.

[0115] As shown in Figure 9, the information processing device 10A includes, as functional units, a decoding unit 11, an acquisition unit 12, an adjustment unit 13, a processing unit 14, a correction unit 15, and a delay unit 16. The functional units of the information processing device 10A can be realized by the processor (CPU (Central Processing Unit), etc.) (not shown) of the information processing device 10A executing a predetermined program using memory (not shown).

[0116] The decoding unit 11, acquisition unit 12, adjustment unit 13, processing unit 14, and correction unit 15 of the information processing device 10A are the same as those provided in the information processing device 10 of this embodiment. The delay unit 16 will be described below.

[0117] The delay unit 16 performs delay processing to delay the sound signals included in the stream. More specifically, when the correction unit 15 delays the change in position or attitude indicated in the second position and attitude information through correction, the delay unit 16 performs delay processing to delay the sound signals by the delayed time (also referred to as delay time). The delay unit 16 also applies suppression processing to subsequent signals, which are sound signals that come after the delayed sound signal, to suppress the delay caused by the delay processing (or to recover the delay caused by the delay processing).

[0118] Delay processing and suppression processing can be performed by well-known speech rate conversion techniques. According to speech rate conversion techniques, the playback speed can be changed without changing the pitch of the sound being reproduced (see Non-Patent Literature 1).

[0119] The delay processing performed by the delay unit 16 will be explained with reference to Figure 10.

[0120] Figure 10 is an explanatory diagram showing the change in yaw angle and the delay of the sound signal in this modified example.

[0121] Figure 10(a) shows the temporal change in the yaw angle 60 of the user U's head and the temporal change in the yaw angle 61 after correction by the correction unit 15.

[0122] The correction unit 15 corrects the yaw angle ψ2 acquired by the acquisition unit 12 at time T12 so that it becomes the yaw angle at time T12A, which is delayed by time L2 from time T12. Similarly, the correction unit 15 corrects the yaw angle ψ3 acquired by the acquisition unit 12 at time T13 so that it becomes the yaw angle at time T13A, which is delayed by time L3 from time T13. Note that the yaw angles ψ1 and ψ4 acquired by the acquisition unit 12 at time T11 and time T14 are not changed by the correction and remain the same before and after the correction.

[0123] Figure 10(b) shows the audio signals included in the stream. Specifically, Figure 10(b) shows, as examples of audio signals included in the stream, audio signal 71 to be played at time T11, audio signal 72 to be played at time T12, audio signal 73 to be played at time T13, and audio signal 74 to be played at time T14. Note that the stream may also include audio signals at times other than those listed above.

[0124] Figure 10(c) shows the sound signals after delay processing or suppression processing by the delay unit 16. Specifically, Figure 10(c) shows the sound signal 71A to be played at time T11, the sound signal 72A to be played at time T12, the sound signal 73A to be played at time T13, and the sound signal 74A to be played at time T14.

[0125] The sound signal 71A is the same as the sound signal 71 before correction. This is because the sound signal 71 has not been corrected by the correction unit 15.

[0126] The sound signal 72A is obtained by delaying the uncorrected sound signal 72 so that it is played back at time T12A with a time delay of L2 from time T12. This is because the delay unit 16 performs delay processing on the sound signal 72 based on the correction unit 15 correcting the yaw angle ψ2 at time T12 so that it becomes the yaw angle at time T12A, which is delayed by time L2 from time T12.

[0127] The sound signal 73A is obtained by delaying the uncorrected sound signal 73 so that it is played back at time T13A, with a delay from time T13. This is because the delay unit 16 performs delay processing on the sound signal 73 based on the correction unit 15 correcting the yaw angle ψ3 at time T13 so that it becomes the yaw angle at time T13A, which is delayed by time L3 from time T13.

[0128] The sound signal 74A is the same as the sound signal 74 before correction. This is because the sound signal 74 has not been corrected by the correction unit 15.

[0129] Thus, during the period P2 in which the delay time tends to increase, the delay unit 16 adds a delay to the sound signal by gradually increasing the delay time. This corresponds to the sound signal being played back slowly.

[0130] Furthermore, during period P3, when the delay time tends to decrease, the delay unit 16 adds a delay to the audio signal while gradually reducing the delay time. This corresponds to the audio signal being played back at a faster speed.

[0131] Furthermore, the delay unit 16 does not perform delay processing or suppression processing during periods P1 and P4 when the correction unit 15 does not apply correction to the sound signal.

[0132] Figure 11 is a flowchart showing the processing of the information processing device 10A in this modified example.

[0133] Steps S101 to S103 are the same as the steps of the same name in the embodiment.

[0134] In step S103A, the delay unit 16 performs delay processing on the sound signal. If the delay unit 16 has already applied delay processing to the sound signal, it applies suppression processing to the subsequent sound signal, which is a sound signal that comes after the delayed sound signal, to suppress the delay caused by the delay processing.

[0135] In step S104, the processing unit 14 uses the position or orientation after delay processing or suppression processing in step S103A to perform stereophonic processing on the sound signal, thereby generating and outputting a sound signal that the speaker should output. The output sound signal is expected to be transmitted to the speaker, output as sound, and heard by user U.

[0136] As a result, the information processing device 10A can suppress the difficulty in hearing the content of the sound output by the sound source, and also suppress the time difference in the timing of the sound signal to which spatial sound processing is applied.

[0137] As described above, the information processing device in the above embodiment or its modified form performs stereophonic sound processing using the corrected position or posture of the user's head, thereby suppressing relatively large changes in the sound heard by the user, which may occur when there is a relatively large change in the position or posture of the user's head. This suppresses large changes in the position of the sound source that the user hears and recognizes, making it easier for the user to hear the content of the sound output by the sound source. Thus, the above information processing method can suppress the difficulty in hearing the content of the sound output by the sound source.

[0138] Furthermore, the information processing device corrects the position or posture information so that the rate of change of the user's head position or posture relative to the sound source becomes the threshold when the rate of change exceeds a threshold, thereby keeping the rate of change of the user's head position or posture relative to the sound source below the threshold. As a result, it is possible to suppress relatively large changes in the sound heard by the user that may occur when there is a large change in the user's head position or posture that exceeds a predetermined standard. In this way, the above information processing method can suppress the difficulty in hearing the content of the sound output by the sound source.

[0139] Furthermore, the information processing device corrects the rate of change in the user's head position or posture relative to the sound source to slow down the change if the rate of change exceeds a threshold, thereby keeping the rate of change in the user's head position or posture relative to the sound source below the threshold. As a result, it is possible to suppress relatively large changes in the sound heard by the user that may occur when there is a large change in the user's head position or posture that exceeds a predetermined standard. In this way, the above information processing method can suppress the difficulty in hearing the content of the sound output by the sound source.

[0140] Furthermore, the information processing device can easily obtain the rate of change of the position or posture change rate using the second derivative of the user's head position or posture with respect to the sound source, and can appropriately correct the user's head position or posture using this rate of change. Therefore, according to the above information processing method, the difficulty in hearing the content of the sound output by the sound source can be suppressed more easily.

[0141] Furthermore, the information processing device uses a smaller threshold for correction in the spatial audio processing of human speech, thereby suppressing large changes in the speed of changes in the user's head position or posture relative to the sound source, especially for speech. Therefore, according to the above information processing method, the difficulty in understanding the content of human speech output by the sound source can be further suppressed.

[0142] Furthermore, the information processing device uses a larger threshold for correction in 3D audio processing of non-human speech, thereby allowing for larger changes in the speed of changes in the user's head position or posture relative to the sound source, and thus reducing the delay in changes in the user's head position or posture. Compared to human speech, there is an advantage in being able to reduce the delay in 3D audio processing when there is less need to make the content of non-human speech easier to hear. Therefore, according to the above information processing method, it is possible to suppress the difficulty in hearing the content of the sound output by the sound source while suppressing the delay in 3D audio processing.

[0143] Furthermore, since the information processing device does not perform corrections in spatial audio processing for non-human speech, there is no delay in changes in the user's head position or posture. Compared to human speech, there is an advantage in being able to further reduce the delay in spatial audio processing when there is less need to make the content of non-human speech easier to hear. Therefore, according to the above information processing method, it is possible to suppress the difficulty in hearing the content of the sound output by the sound source while suppressing the delay in spatial audio processing.

[0144] Furthermore, the information processing device delays the sound signal by a delay time that delays the change in position or posture indicated in the second position and posture information through correction. This suppresses the time difference that may occur between the stereophonic sound processing based on the user's head position or posture and the sound signal to which the stereophonic sound processing is to be applied. Therefore, according to the above information processing method, the difficulty in hearing the content of the sound output by the sound source can be further suppressed.

[0145] Furthermore, the information processing device contributes to recovering the delay in the sound signal caused by the delay processing through suppression processing. Therefore, according to the above information processing method, the difficulty in hearing the content of the sound output by the sound source can be further suppressed.

[0146] In the above embodiment, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, the software that implements the information processing device, etc., of the above embodiment is the following program.

[0147] In other words, this program causes a computer to execute an information processing method that acquires a stream including first position and orientation information indicating the position and orientation of a sound source and an audio signal indicating the sound output by the sound source, acquires second position and orientation information indicating the position and orientation of the user's head, and performs a correction to reduce the rate of change of the change rate of the position or orientation of the sound source indicated in the acquired second position and orientation information with respect to the position or orientation of the sound source indicated in the first position and orientation information, thereby acquiring the second position and orientation information used for stereophonic sound processing of the audio signal using the first position and orientation information and the second position and orientation information.

[0148] Although the present invention has been described above based on embodiments of one or more information processing devices, the present invention is not limited to these embodiments. Without departing from the spirit of the present invention, various modifications that a person skilled in the art can conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments. [Industrial applicability]

[0149] This invention can be used in an information processing device that performs stereophonic sound processing. [Explanation of Symbols]

[0150] 5 Sound Sources 10, 10A Information Processing Device 11 Decoding section 12 Acquisition Department 13 Adjustment part 14 Processing Unit 15 Correction section 16 Delay section 30, 31, 32 angle range 51, 52, 53, 54, 55, 56 waveform 60, 61, 62 yaw angles 71, 71A, 72, 72A, 73, 73A, 74, 74A sound signal P1, P2, P3, P4 period S space U User

Claims

1. An information processing device comprising a processor and memory, The processor uses the memory to: A stream is acquired that includes first position and orientation information indicating the position and orientation of the sound source, and an audio signal indicating the sound output by the sound source. Secondary position and orientation information indicating the position and orientation of the user's head is obtained. The updated position or posture of the user's head, based on the acquired second position and posture information, is generated as a temporally interpolated position or posture. Using the first position and orientation information and the temporally interpolated second position and orientation information, stereophonic processing is performed on the sound signal. Information processing device.

2. The temporally interpolated second position and orientation information is generated over the period corresponding to the update. The information processing apparatus according to claim 1.

3. The interpolation is performed so as to the temporal change in the acquired second position and orientation information. The information processing apparatus according to claim 1 or 2.

4. The aforementioned spatial audio processing is performed based on the relative position or orientation between the sound source and the user's head. The information processing apparatus according to any one of claims 1 to 3.

5. The aforementioned spatial audio processing is performed by calculating the arrival timing of at least one of the direct sound and reflected sound arriving from the sound source to the user. The information processing apparatus according to any one of claims 1 to 4.

6. The interpolation described above is not performed under predetermined conditions. The information processing apparatus according to any one of claims 1 to 5.

7. The stream further includes classification information indicating whether the sound indicated by the sound signal is a human voice or not. The interpolation is not performed if the type information indicates that the sound signal is not a human voice. The information processing apparatus according to any one of claims 1 to 6.

8. An information processing method performed by an information processing device equipped with a processor and memory, The processor uses the memory, A stream is acquired that includes first position and orientation information indicating the position and orientation of the sound source, and an audio signal indicating the sound output by the sound source. Secondary position and orientation information indicating the position and orientation of the user's head is obtained. The updated position or posture of the user's head, based on the acquired second position and posture information, is generated as a temporally interpolated position or posture. Using the first position and orientation information and the temporally interpolated second position and orientation information, stereophonic processing is performed on the sound signal. Information processing methods.

9. A program that causes an information processing device to execute the information processing method described in claim 8.

Images