Speech processing device and program

Abstract

To provide a voice processing device and a program for allowing users to more easily listen to an object to which a creator wants the users to listen.SOLUTION: In a speech processing device 10, a control unit 110 for rendering an acquired sound signal to synthesize a sound for playback and outputting the synthesized sound to a playback unit 130 renders a sound for each object according to a sound source direction based on a listening position. A masking analysis unit includes an output processor for calculating the amount of masking for the sound of a first type object by the sound of a second type object such that the amount of masking decreases as the angle between a first direction representing a sound source direction of the first type object and a second direction representing a sound source direction of the second type object increases, and a volume adjustment unit for adjusting the level of the sound of the second type object such that the level difference between the level of the sound of the first type object and the amount of masking is equal to or more than a predetermined reference level difference when the level difference is smaller than a predetermined reference level difference.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present application relates to an audio processing device and a program, and, for example, to adjusting the volume of an object in audio rendering according to the positional relationship between a sound source and a listener. [Background technology]

[0002] 6DoF (Degrees of Freedom) content is composed of images and sounds that can be viewed from various positions or directions, and may be applied to Augmented Reality (AR) or Virtual Reality (VR) (collectively referred to as "AR / VR" in this application). In AR / VR content, in order to improve the sense of reality and immersion, it has been proposed to play back sound according to the positional relationship between the sound source and the listener, taking into account the radiation characteristics of the sound source reproduced in three-dimensional space (Non-Patent Documents 1-3).

[0003] Content based on normal channel-based audio is produced by adjusting the way sounds are heard at a specific listening position. In contrast, 6DoF content allows users to freely set their listening position. It is not realistic to adjust the way sounds are heard for every listening position and listening direction in the content space. There is a risk that sound quality will not be guaranteed for listening positions or directions that the creator did not anticipate, as the way sounds are heard will not be adjusted.

[0004] In general, when sound waves emitted from a sound source are blocked by another object, the diffracted waves that propagate by diffracting the surface of the object are attenuated more significantly as their wavelengths become shorter and their high-frequency components become shorter. Let us assume that a sound radiation characteristic of an object is given to an object that the creator wants the listener to hear. In that case, it is expected that the rendered sound will be difficult to hear at a listening position set behind the object due to the significant attenuation of high-frequency components.

[0005] In cases where dialogue sounds such as lines are difficult to hear in broadcasts, it has been proposed to adjust the volume level of the dialogue sounds independently of the background sounds (dialogue enhancement) (Non-Patent Document 4). The dialogue enhancement function adjusts the volume level regardless of the frequency band. To deal with the attenuation of high-frequency components, the volume level of the dialogue sounds is increased until a sufficient level difference between the dialogue and background sounds is obtained for the high-frequency components to be heard. As a result, the volume level of the lower frequency components may be increased more than necessary.

[0006] On the other hand, it has been proposed to apply object-based audio (Non-Patent Documents 7 and 8) to 6DoF content (Non-Patent Document 9). In object-based audio, control information such as the position and gain of an object is described as element information in metadata (Non-Patent Documents 5 and 6), and is referenced when generating an output audio signal for playback from the audio signal of the object. The metadata can also describe information such as the importance of an object and the type of sound source, allowing the creator to specify the object that they want the viewer to hear. The metadata can also set the volume based on the importance of the object. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Kotaro Kinoshita, Takehiro Sugimoto, Kazuho Ono, Study on measurement method of three-dimensional radiation characteristics during Japanese speech, Proceedings of the Acoustical Society of Japan, 2020 Autumn, 1-1-18, August 26, 2020 [Non-Patent Document 2] Kotaro Kinoshita, Takehiro Sugimoto, Yasushige Nakayama, Study on estimation method of three-dimensional radiation characteristics of human voice, Proceedings of the Acoustical Society of Japan, 2021 Autumn, 2-2-12, August 24, 2021 [Non-Patent Document 3] Takehiro Sugimoto, Kotaro Kinoshita, Angular resolution required for reproducing human voice in any three-dimensional radiation direction - Study on horizontal and median planes, Proceedings of the Acoustical Society of Japan Research Presentation Conference, Autumn 2021, 2-8-8, August 24, 2021 [Non-Patent Document 4] Takehiro Sugimoto et al., Multifunctionality of 22.2 Multichannel Sound in Broadcasting Services, AES Japan Conference Nagoya 2015, September 11, 2015 [Non-Patent Document 5] Recommendation ITU-R BS.2076-2, Audio definition model, 10 / 2019 [Non-Patent Document 6] Recommendation ITU-R BS.2125-0, A serial representation of the Audio Definition Model, 01 / 2019 [Non-Patent Document 7] Recommendation ITU-R BS.2127-0, Audio Definition Model renderer for advanced sound systems, 06 / 2019 [Non-Patent Document 8] ISO / IEC 23008-3:2019, Information technology - High efficiency coding and media delivery in heterogeneous environments - Part3: 3D Audio, 2019-02 [Non-Patent Document 9] Immersive experience of programs using AR technology, Giken Open House 2021, Japan Broadcasting Corporation, 2021 / 6 / 1 - 2021 / 6 / 30, [Retrieved August 10, 2023], Internet<URL:https: / / www.nhk.or.jp / strl / open2021 / tenji / 2 / > [Non-Patent Document 10] Atsunobu Murase, Kazuhiro Nakamura, Kazuhiro Iida, Sound quality control method considering masking by ambient noise, Proceedings of the Acoustical Society of Japan, Spring 1997, p.523-524, March 1997 [Non-Patent Document 11] Atsunobu Murase, Kazuhiro Nakamura, Kazuhiro Iida, Improvement of Sentence Intelligibility by Noise-Adaptive Public Address System, Proceedings of the Acoustical Society of Japan, 2000(2), p.459-460, 2000-09-01 [Non-Patent Document 12] Nishiguchi Masayuki, et al., Spatial Auditory Masking for Three-Dimensional Audio Coding, AES Convention 147, Paper 10259, October 8, 2019 [Non-Patent Document 13] Tomoyasu Komori, Setsu Komiyama, Hiroya Dan, A study on audio mixing balance using loudness as an index, Technical Report of the Institute of Electronics, Information and Communication Engineers, 107(432), p.37-42, 2008-01-24 [Non-Patent Document 14] Torcoli, Matteo et al., Background Ducking to Produce Esthetically Pleasing Audio for TV with Clear Speech, AES Convention 146, Paper 10175, March 10, 2019 Summary of the Invention [Problem to be solved by the invention]

[0008] In object-based audio, the listener may set the listening position arbitrarily. This may increase the possibility that the object that the producer wants the listener to hear may be difficult to hear. For example, the volume of the sound of an object that is far from the listening position decreases due to distance attenuation. Furthermore, when the sound of another object is played back, the sound may be masked and drowned out.

[0009] The embodiments of the present application have been made to solve the above problems, and one of the objectives of the application is to provide an audio processing device and program that make objects that the creator wants people to hear easier to hear. [Means for solving the problem]

[0010] [1] One aspect of this embodiment is an audio processing device that includes an output processing unit that renders audio for each object according to a sound source direction based on a listening position; an analysis unit that calculates an amount of masking of the audio of the second type of object by the audio of the first type of object so that the amount of masking becomes smaller as an angle between a first direction that is the sound source direction of a first type of object and a second direction that is the sound source direction of a second type of object increases; and an adjustment unit that adjusts the level of the audio of the second type of object so that, when a level difference between the level of the audio of the first type of object and the masking amount is smaller than a predetermined reference level difference, the level difference becomes equal to or greater than the predetermined reference level difference. According to the configuration of [1], when the angle between the first direction and the second direction becomes smaller, the amount of masking by the second type object increases, and the level difference between the level of the sound of the first type object and the amount of masking by the second type object becomes smaller than the reference level difference, the level of the sound of the second type object is lowered so that the level difference becomes equal to or greater than the reference level difference. Therefore, when the sound of the second type object is reproduced, the sound of the first type object can be clearly heard by the listener.

[0011] [2] One aspect of this embodiment is the above-mentioned audio processing device, wherein the analysis unit may calculate the masking amount based on the smaller of a first angle, which is the angle between the first direction and the second direction, and a second angle, which is the angle between the second direction and an inverted direction, which is a direction inverted forward and backward from the first direction. According to the configuration of [2], the smaller of the first and second angles is the angle between the left and right directions obtained without distinguishing between the front and rear of the first and second directions, and the masking amount is calculated based on that angle. Therefore, even if one or both of the sound of the first object and the second object are perceived in a direction that is inverted from the target direction, the masking amount according to the angle between the perceived directions is obtained.

[0012] [3] One aspect of this embodiment is the audio processing device described above, wherein when the number of the second type objects is multiple, the adjustment unit may calculate a weighted sum of the individual masking amounts of the first type objects for each of the second type objects as the masking amount for the entire second type objects using a weighting coefficient that is smaller the larger the angle, and adjust the level of the audio for each of the second type objects based on the level difference. According to the configuration of [3], the masking amount of the entire second type object is calculated so that the contribution of the second type object is smaller as the angle between the second direction and the first direction becomes larger. Since a value closer to the actual measured value is calculated than when the contributions of the multiple second type objects are uniform, the sound level of the second type object is adjusted based on the calculated value, and the sound of the first type object can be clearly heard without lowering the sound level of the second type object more than necessary.

[0013] [4] One aspect of this embodiment is the above-mentioned audio processing device, wherein the analysis unit calculates the masking amount for each predetermined frequency band, and the adjustment unit adjusts the level of the audio of the second type object for each frequency band based on a level difference between the level of the audio of the first type object and the masking amount. According to the configuration of [4], the amount of masking is calculated for each frequency band, and the level of the sound of the second type object is adjusted based on the level difference between the level of the sound of the first type object and the calculated amount of masking. Therefore, the level of the sound of the second type object is adjusted taking into consideration the presence or absence of band components of the first type object for each frequency band and the characteristics of masking that depend on the frequency band.

[0014] [5] One aspect of the present embodiment may be a program for causing a computer to function as the above-described voice processing device. According to the configuration of [5], when the angle between the first direction and the second direction becomes smaller, the amount of masking by the second type object increases, and the level difference between the level of the sound of the first type object and the amount of masking by the second type object becomes smaller than the reference level difference, the level of the sound of the second type object is lowered so that the level difference becomes equal to or greater than the reference level difference. Therefore, when the sound of the second type object is played back, the sound of the first type object can be clearly heard by the listener. Effect of the Invention

[0015] According to the above configuration, it is possible to make an object that the creator wishes to be heard more audibly. [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic block diagram for explaining an example of a functional configuration of a sound processing device according to an embodiment of the present invention; [Diagram 2] 13 is a flowchart illustrating an example of a level control process for a non-essential object according to the embodiment. [Diagram 3] 10 is a flowchart illustrating a level difference calculation process according to the present embodiment. [Figure 4] 11 is a plan view showing an example of an azimuth angle for each object according to the embodiment; FIG. [Diagram 5] 11A and 11B are side views showing examples of elevation angles for each object according to the embodiment. [Figure 6] 11A and 11B are plan views showing examples of azimuth angles and inversion direction azimuth angles for each object according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] (overview) Hereinafter, an embodiment of the present application will be described with reference to the drawings. First, an overview of the present embodiment will be described. Fig. 1 is a schematic block diagram for explaining an example of the functional configuration of a voice processing device 10 according to the present embodiment. The audio processing device 10 includes a control unit 110 , a playback unit 130 , a display unit 140 , and an input unit 150 .

[0018] The control unit 110 acquires content that includes audio as a component. The control unit 110 performs rendering on the acquired audio signal to synthesize audio for playback. The control unit 110 outputs output audio data indicating the synthesized audio to the playback unit 130. The content is object-based content that includes one or more objects. The acquired content includes audio data and metadata. The objects correspond to the material of the content and correspond to individual objects that are sound sources. The objects may include people such as performers, and animals and plants. The audio data is data that includes an audio signal for each object. Various parameters are described in the metadata for each object.

[0019] The described parameters include attributes of the object and control information related to the presentation of the object. The attributes include, for example, the type of the object. The control information includes, for example, spatial information of the object. The spatial information includes the position of the object in the content space. The content space is a virtual or real space in which individual objects are placed, and is expressed, for example, with a specific position in the content space or the listening position as the origin. The position of the object includes information on the direction and distance based on the listening position.

[0020] The content may include components of a type other than audio, such as video. The content may include, for example, video data including a video signal for each object. In this case, the control unit 110 synthesizes a video for playback by performing a rendering process on the acquired content. The control unit 110 may output output video data indicating the synthesized video to the display unit 140.

[0021] In this application, "viewing" may refer to both seeing and listening, or to either one. The direction of an object that is a sound source may be called the "sound source direction" and its position may be called the "sound source position." A user of the sound processing device 10 may be called a "listener" or alternatively called a "viewer." The "listening position" and the "listening direction" may be interchangeably called the "viewing position" and the "listening direction," respectively. The listening direction is, for example, a reference direction that corresponds to the front direction of the listener's head.

[0022] Objects are classified as required objects and non-required objects. Required objects are objects that should be presented to listeners with priority. In other words, required objects are objects that the creator wants listeners to listen to without fail. Non-required objects are objects other than required objects. Whether or not an object is required is indicated for each object using a required flag. The required flag is a type of parameter included in metadata and is set by the content creator.

[0023] The control unit 110 calculates the masking amount of the audio of the mandatory object by the audio of the non-mandatory object so that the masking amount decreases as the angle between the first direction, which is the target direction of the mandatory object, and the second direction, which is the target direction of the non-mandatory object, increases. The control unit 110 determines whether the level difference between the level of the audio of the mandatory object and the calculated masking amount is equal to or greater than a predetermined reference level difference. When the calculated level difference is smaller than the reference level difference, the control unit 110 adjusts the level of the audio signal of the non-mandatory object so that the level difference with the masking amount is equal to or greater than the reference level.

[0024] The reproduction unit 130 reproduces sound based on the output audio data input from the control unit 110. The reproduction unit 130 includes a reproduction sound source corresponding to a predetermined sound reproduction method. When the reproduction method is multi-channel sound, the reproduction unit 130 includes a plurality of speakers. Each speaker is associated with a reproduction channel. The plurality of speakers are arranged at predetermined positions defined by the reproduction method. Examples of the multi-channel sound include reproduction methods such as 22.2ch sound, 7.1ch surround, and 5.1ch surround. The arrangement of speakers in 22.2ch sound is defined in ITU-R Recommendation BS.2051. When the reproduction method is binaural sound, the reproduction unit 130 includes two reproduction sound sources. The reproduction unit 130 may be configured to be wearable on the head of a viewer, such as headphones or earphones.

[0025] The display unit 140 displays an image based on the output image data input from the control unit 110. The display unit 140 includes, for example, a display monitor. The display monitor may be, for example, any of an OLED (Organic Light Emitting Diode) display, an LCD (Liquid Crystal Display), and the like. The display unit 140 may be configured to be mountable on the viewer's head as a head mounted display (HMD), a glasses-type display, etc. In this case, the display unit 140 may be configured integrally with the headphones or earphones that constitute the playback unit 130.

[0026] The input unit 150 receives an operation from a listener who is a user, and outputs an operation signal corresponding to the received operation to the control unit 110. The input unit 150 includes members such as a mouse, a touch sensor, a dial, and a lever.

[0027] The audio processing device 10 may include a detection unit (not shown). The detection unit includes a sensor that detects the position (i.e., listening position) and orientation (i.e., listening direction) of the viewer. The detection unit is, for example, a six-axis acceleration sensor. The six-axis acceleration sensor is capable of detecting the position and direction in a three-dimensional space. The detection unit may be installed in one or both of the reproduction unit 130 and the display unit 140 that are configured to be wearable on the viewer's head.

[0028] Next, a description will be given of an example of the functional configuration of the control unit 110. The control unit 110 includes a content acquisition unit 112, a voice presence determination unit 114, a masking analysis unit 116, a volume adjustment unit 118, and an output processing unit 122. The content acquisition unit 112 acquires the content specified by the output processing unit 122. The content acquisition unit 112 includes, for example, a storage medium in which various contents are stored in advance, and reads out the specified content. The content acquisition unit 112 may include a broadcast receiving unit and separate multiplexed content from a broadcast signal obtained by receiving a broadcast wave on a selected broadcast channel. In the content of a broadcast program, metadata may be described in program information that is a part of control information of the broadcast program. The content acquisition unit 112 may include a communication unit and receive the content specified from a content server connected to a communication network. The content acquisition unit 112 outputs the acquired content to the voice presence determination unit 114 and the output processing unit 122.

[0029] The voiced section determination unit 114 extracts a mandatory object from the content input from the content acquisition unit 112, and determines whether the latest section up to the present time is a voiced section for the audio signal of the extracted mandatory object at a predetermined time interval (typically 0.2 to 1 s, for example 0.4 s). For example, the voiced section determination unit 114 calculates a loudness level based on the audio signal, and can determine whether the calculated loudness level is equal to or higher than a predetermined lower limit level. The lower limit level may be a value significantly higher than the background noise level. When the voiced section is determined to be a voiced section, the voiced section determination unit 114 outputs the audio signal and metadata extracted for each object to the masking analysis unit 116. When the section is determined to be an unvoiced section, no audio signal is output. At this time, no processing is performed to adjust the audio level of the non-mandatory content, which will be described later.

[0030] Instead of the loudness level, the voiced sound determination unit 114 may use another volume index, for example, the root mean square (RMS) of the signal value for each sample. The voiced sound determination unit 114 may use a known voiced sound determination method (VAD: Voice Activity Detection) to determine whether or not each section of a predetermined time interval is a voiced section.

[0031] The masking analysis unit 116 divides the audio signal for each object input from the voice presence determination unit 114 into band component signals having band components of audio for each predetermined frequency band (band division). For example, a 1 / 3 octave band is used as the frequency band. The 1 / 3 octave band is approximate to a critical band. The critical band corresponds to the bandwidth of the auditory filter of the human auditory system. In human hearing, when a first sound having band components of a certain frequency band and a second sound having band components of a frequency band overlapping with the first sound are presented, a phenomenon called masking is known in which the second sound (masker) inhibits the first sound (maskey) from being heard. The amount of masking corresponds to the difference between the minimum audible level for the first sound when the second sound is not presented and the minimum audible level for the first sound when the second sound is presented. For example, when a pure tone is presented as a first sound, if band noise with the frequency of the pure tone as its center frequency is presented as a second sound, the first sound is inhibited from being heard. When the bandwidth of the band noise is increased from a narrow bandwidth while keeping the spectrum level constant, the amount of masking for the pure tone of the center frequency increases, but there is a bandwidth beyond which the amount of masking does not increase even if the bandwidth is further increased. This bandwidth corresponds to the critical band.

[0032] The masking analysis unit 116 calculates the amount of masking by the band components of the non-essential object against the band components of the essential object for each frequency band based on a predetermined spatial masking model. The masking analysis unit 116 identifies the sound source position and listening position for each object from the metadata of the content, and determines the direction of the sound source position based on the listening position as the sound source direction. When the sound source position of the object is notified from the output processing unit 122, the notified sound source position may be used instead of the sound source position described in the metadata.

[0033] The masking analysis unit 116 calculates the masking amount using the first direction, which is the sound source direction of the essential object, its band components, and the second direction, which is the sound source direction of the non-essential object, and its band components. According to spatial masking, the calculated masking amount tends to decrease as the angle between the first direction and the second direction increases. In addition, the signal level n of the band components of the non-essential object m is the signal level s of the band component of the required object m The masking amount increases as the signal level s of the band component of the essential object for each frequency band is increased. m Masking amount M m The difference between these is notified to the volume adjustment unit 118 as a level difference. Note that m is an index indicating a frequency band. m is, for example, an integer value between 1 and M (M is an integer indicating the number of divisions of the frequency band) and is set in ascending order of frequency.

[0034] For example, when the azimuth angle θ' in the second direction is within a range of a predetermined azimuth angle width Δθ (typically, 30° to 60°, e.g., 45°) from the azimuth angle θ in the first direction (θ-Δθ<θ'<θ+Δθ) and the elevation angle φ' in the second direction is within a range of a predetermined elevation angle width Δφ (typically, 30° to 45°, e.g., 45°) from the elevation angle φ in the first direction (φ-Δφ<φ'<φ+Δφ), the masking analysis unit 116 calculates an amount of masking M of the band components of the essential object by the band components of the non-essential object in the mth band using formula (1). m (dB) is calculated (see Non-Patent Documents 10 and 11). In the formula (1), s m indicates the signal level of the band component of the required object in the mth band. m indicates the signal level of the band component of the non-essential object in the mth band.

[0035]

number

[0036] In response to this, when the azimuth angle θ' in the second direction is outside the range of a predetermined azimuth angle width Δθ from the azimuth angle θ in the first direction (θ-Δθ≧θ' or θ+Δθ≦θ'), or when the elevation angle φ' in the second direction is outside the range of a predetermined elevation angle width Δφ from the elevation angle φ in the first direction (φ-Δφ≧φ' or φ+Δφ≦θ'), the masking analysis unit 116 calculates an amount of masking M m (dB) is calculated by the masking amount M given by equation (2). m is the masking amount M given by equation (1). m This is because when the angle between the maskee and the masker is 45°, the amount of masking is 3 dB lower than when the angle is 0° (see Non-Patent Document 12).

[0037]

number

[0038] The volume adjustment unit 118 judges whether or not the level difference for each frequency band notified by the masking analysis unit 116 is equal to or greater than a predetermined reference level difference (typically 5 to 10 [dB], for example 7 [dB]). For frequency bands in which the level difference is equal to or greater than the reference level difference, the volume adjustment unit 118 does not perform level adjustment on the band components (audio signals) of the non-essential objects. The volume adjustment unit 118 determines the gain for the band components of the non-essential objects in that frequency band to be 1 (corresponding to 0 [dB]). This gain corresponds to a level adjustment value.

[0039] For a frequency band in which the level difference is less than the reference level difference, the volume adjustment unit 118 adjusts the volume level of the band component of the non-essential object so that the level difference from the masking amount becomes equal to or greater than the reference level difference. For example, the volume adjustment unit 118 adjusts the signal level s of the band component of the essential object. m and the masking amount M calculated using equations (1) and (2). mA gain for the band components of the non-essential object is determined so that the level difference with the masking amount is set to a reference level. Since the band components of the non-essential object are reduced and the level difference with the masking amount increases, the gain is determined to a positive integer value smaller than 1. The volume adjustment unit 118 outputs level adjustment information indicating the determined gain to the output processing unit 122. This allows level adjustment to be performed taking into account the spatial masking effect.

[0040] The number of non-essential objects is not limited to one, and may be multiple. A different second direction is set for each of the multiple non-essential objects, and the angle between the second direction and the first direction may be different. The masking analysis unit 116 calculates the masking amount for the essential object for each non-essential object in each frequency band, and calculates the sum of the calculated masking amounts in the linear region as the masking amount M of all the non-essential objects. m The masking amount M m The process of determining can also be considered as a weighted sum according to the angle between the first direction and the second direction.

[0041] In the above, the volume adjustment unit 118 adjusts the level difference between the audio level of the required object and the masking amount of the non-required object so as to be equal to or greater than a predetermined reference level difference, but the user can adjust it freely. In this case, the masking amount is weighted according to the relationship between the first direction and each second direction, so that adjustment can be made taking spatial masking into consideration. The weighting coefficient is instructed, for example, by an operation signal from the input unit 150. In the example of equations (1) and (2), the volume adjustment unit 118 determines the weighting coefficient to be 1 when the azimuth angle θ' of the second direction is within a range of a predetermined azimuth angle width Δθ from the azimuth angle θ of the first direction, and the elevation angle φ' of the second direction is within a range of a predetermined elevation angle width Δφ from the elevation angle φ of the first direction. The volume adjustment unit 118 determines the weighting factor to be 1 / √2 when the azimuth angle θ' of the second direction is outside the range of a predetermined azimuth angle width Δθ from the azimuth angle θ of the first direction, or when the elevation angle φ' of the second direction is outside the range of a predetermined elevation angle width Δφ from the elevation angle φ of the first direction. This weighting factor corresponds to the masking amount given by equation (2) being 3 dB lower than the masking amount given by equation (1).

[0042] When there are a plurality of non-essential objects and the volume levels of the non-essential objects are adjusted, the volume adjustment unit 118 may determine a common gain for the plurality of non-essential objects, or may determine a different gain for each non-essential object. For example, the volume adjustment unit 118 may determine a gain so that the gain is smaller for a non-essential object having a smaller difference between the volume level of the essential object and the amount of masking of each non-essential object with respect to the essential object.

[0043] The output processing unit 122 identifies the content designated by the operation signal input from the input unit 150, and causes the content acquisition unit 112 to acquire the identified content. The output processing unit 122 may, for example, cause the display unit 140 to display a setting screen showing a list showing the names or locations of the content that can be acquired. The location of the content is represented, for example, by a broadcast channel showing the carrier frequency of the broadcast wave, or by a URL (Universal Resource Locator). The URL is an address showing the location on a device connected to a communication network or on a specific device.

[0044] The output processing unit 122 synthesizes audio for playback based on audio data included in the content acquired from the content acquisition unit 112. The output processing unit 122 outputs output audio data indicating the audio for playback to the playback unit . The output processing unit 122 refers to metadata included in the content and extracts an audio signal for each object from the audio data. The output processing unit 122 divides the audio signal of the non-essential object into band component signals having band components for each predetermined frequency band. The output processing unit 122 adjusts the amplitude by multiplying the divided band component signals by the gain of the frequency band input from the volume adjustment unit 118. When the gain is determined for each non-essential object, the output processing unit 122 identifies a non-essential object to be adjusted corresponding to that gain, and multiplies the band component signal related to the identified non-essential object by that gain. Then, the output processing unit 122 synthesizes the band component signals after the amplitude adjustment between frequency bands to generate an audio signal after the amplitude adjustment.

[0045] The output processing unit 122 adds acoustic characteristics corresponding to spatial information of the object specified by referring to the metadata to the audio signal for each object (the audio signal after amplitude adjustment for non-essential objects) to generate an individual output audio signal (rendering). The output processing unit 122 adds (mixes) the generated individual output audio signals between objects to synthesize output audio signals. The output processing unit 122 outputs output audio data indicating the output audio signal to the playback unit 130.

[0046] The process of rendering for the sound source direction depends on the playback method of the playback unit 130. When the playback method is multi-channel audio, playback control data indicating the volume ratio between playback channels for each sound source direction is set in advance in the output processing unit 122. The output processing unit 122 refers to the playback control data and specifies the volume ratio corresponding to the sound source direction for each object. The output processing unit 122 further adjusts the amplitude for each playback channel according to the specified volume ratio for the audio signal after the amplitude adjustment, and further distributes the amplitude-adjusted audio signal to generate a multi-channel output audio signal for each object.

[0047] When the playback method is binaural sound, playback control data indicating the head-related transfer function for each playback channel for each sound source direction is set in advance. In binaural sound, the playback channels include a left channel corresponding to the left ear and a right channel corresponding to the right ear. The output processing unit 122 refers to the playback control data and specifies the head-related transfer function for each playback channel corresponding to the sound source direction for each object. The output processing unit 122 performs a convolution operation on the audio signal after the amplitude adjustment based on the head-related transfer function specified for each playback channel to generate an output audio signal for each object.

[0048] The output processor 122 then adjusts the amplitude of the object-specific output audio signal for each sound source direction according to a predetermined distance attenuation law based on the distance from the listening position to the sound source position to generate an individual output audio signal. As the distance attenuation law, for example, the inverse square law can be applied.

[0049] Note that the position of the object may be variable. For example, an object variable flag indicating whether or not the position of each object is variable may be included in the metadata, or may be set in the output processing unit 122. In this case, the output processing unit 122 can refer to the object variable flag to determine whether or not the position of the object is variable. When the position of the object is variable, the output processing unit 122 employs the position of the object designated by an operation signal input from the input unit 150, instead of the position of the object indicated in the metadata.

[0050] Furthermore, the listening position may be variable. For example, a listening position variable flag indicating whether or not the listening position is variable may be included in the metadata, or may be set in the output processing unit 122. In this case, the output processing unit 122 can refer to the listening position variable flag and determine whether or not the listening position is variable. When the listening position is variable, the output processing unit 122 employs a listening position designated by an operation signal input from the input unit 150. The output processing unit 122 may cause the display unit 140 to display a setting screen indicating the positions of individual objects (corresponding to sound source positions) and the listening positions in the content space.

[0051] When the content includes video, the output processing unit 122 synthesizes a video for display based on video data included in the content acquired from the content acquisition unit 112. The output processing unit 122 outputs output video data indicating the video for display to the display unit 140. For example, the output processing unit 122 refers to metadata and identifies spatial information for each object. The output processing unit 122 uses a known image processing method to synthesize an output image observed from the viewing position based on the image data included in the acquired content (image rendering). For example, volumetric video can be used as the image processing method. In volumetric video, the output processing unit 122 synthesizes, as individual images, images observed within a field of view centered on the viewing direction from the viewing position with respect to the three-dimensional shapes of the background and objects in the content space shown in the image data. The output processing unit 122 can synthesize an output image by superimposing individual images showing each object on the individual image showing the background.

[0052] Next, an example of the level control process of the non-essential object according to this embodiment will be described. Fig. 2 is a flowchart showing an example of the level control process of the non-essential object according to this embodiment. The process in Fig. 2 is executed at predetermined time intervals. (Step S102) The voiced section determination unit 114 calculates the volume at a predetermined time interval for the required object extracted from the content acquired from the content acquisition unit 112, and determines whether the section is a voiced section or not based on the calculated volume. If it is determined to be a voiced section (step S102 YES), the process proceeds to step S104. If it is determined to be an unvoiced section (step S102 NO), the process of FIG. 2 ends.

[0053] (Step S104) The masking analysis unit 116 calculates, for each frequency band, the amount of masking by the band components of the non-essential object with respect to the band components of the essential object so that the amount decreases as the angle between the first direction and the second direction increases. (Step S106) The masking analysis unit 116 calculates a level difference between the level of the band component of the essential object and the amount of masking by the band component of the non-essential object.

[0054] (Step S108) The volume adjustment unit 118 judges whether or not the calculated level difference is less than a predetermined reference level difference. If it is judged to be less than the reference level difference (step S108 YES), the process proceeds to step S110. If it is judged to be equal to or greater than the reference level difference (step S108 NO), the process of FIG. 2 ends. (Step S110) The volume adjustment unit 118 determines a gain for adjusting the level of the band component of the non-essential object so that the level difference between the level of the band component of the essential object and the amount of masking by the band component of the non-essential object on the band component of the essential object is equal to or greater than a reference level difference. The volume adjustment unit 118 outputs the determined level to the output processing unit 122, which adjusts the volume level of the band component of the non-essential object based on the gain determined for that frequency band. Then, the process of FIG. 2 ends.

[0055] It is known that front-back errors occur in the perception of sound direction. A front-back error is an event in which a sound image is perceived in a direction that is reversed from the actual sound source direction or the sound source direction specified by the content creator (sometimes referred to as an "inverted direction" in this application). In this way, the uncertainty of front-back perception is also reflected in spatial masking. That is, the smaller the angle between the reverse direction of the masker's sound source direction and the maskee's sound source direction, the greater the amount of masking tends to be.

[0056] Therefore, the masking analysis unit 116 may calculate the masking amount by the band component of the non-essential object for the band component of the essential object, further taking into consideration the inversion direction that is inverted front-to-back with respect to the first direction. That is, the masking analysis unit 116 calculates the masking amount so that the smaller of the first angle, which is the angle between the first direction and the second angle, and the second angle, which is the angle between the inversion direction that is inverted front-to-back with respect to the first direction, is larger, the smaller the masking amount becomes. In this way, the masking amount that takes into account the possibility of front-to-back errors is estimated.

[0057] In the example of FIG. 6 , when the azimuth angle θ' of the second direction is within a range of a predetermined azimuth angle width Δθ from the azimuth angle θ of the first direction (θ-Δθ<θ'<θ+Δθ), or the azimuth angle θ' of the second direction is within a range of a predetermined azimuth angle width Δθ from the azimuth angle -θ of the opposite direction to the first direction (-θ-Δθ<θ'<-θ+Δθ) and the elevation angle φ' of the second direction is within a range of a predetermined elevation angle width Δφ from the elevation angle φ of the first direction (φ-Δφ<φ'<φ+Δφ), the masking analysis unit 116 calculates an amount of masking M of the band components of the non-essential object in the m-th band by the band components of the essential object. m Calculate (dB).

[0058] When the azimuth angle θ' of the second direction is outside the range of a predetermined azimuth angle width Δθ from the azimuth angle θ of the first direction (θ-Δθ≧θ' or θ'≧θ+Δθ), and when the azimuth angle θ' of the second direction is outside the range of a predetermined azimuth angle width Δθ from the azimuth angle -θ of the opposite direction to the first direction (-θ-Δθ≧θ' or θ'≧-θ+Δθ), or when the elevation angle φ' of the second direction is outside the range of a predetermined elevation angle width Δφ from the elevation angle φ of the first direction (φ-Δφ≧φ' or φ'≧φ+Δφ), the masking analysis unit 116 calculates an amount of masking M of the band components of the non-essential object in the m-th band by the band components of the essential object. m Calculate (dB).

[0059] Note that front-back errors tend to occur more frequently when binaural sound is used as the playback method than when multi-channel sound is used, so the masking analysis unit 116 applies this technique when the playback method applied to the playback unit 130 is binaural sound, but does not have to apply it when the playback method is multi-channel sound. Also, sequence errors tend to occur more frequently when a video showing an object related to audio playback is not displayed than when it is displayed. Therefore, the masking analysis unit 116 applies this technique when the output processing unit 122 causes the display unit 140 to display a display video showing an object, but does not have to apply this technique when the display video is not displayed.

[0060] Next, an example of the level difference calculation process according to this embodiment will be described. Fig. 3 is a flowchart illustrating the level difference calculation process according to this embodiment. The process in Fig. 3 corresponds to an example of the process in steps S104 and S106 in Fig. 2.

[0061] (Step S202) The masking analysis unit 116 identifies the sound source direction of the required object relative to the listening position as a first direction (θ, φ) based on the metadata of the content or the operation signal, and identifies the sound source direction of the non-required object as a second direction (θ', φ'). (Step S204) Masking analysis unit 116 judges whether the azimuth angle of the second direction is within a predetermined range from the azimuth angle of the first direction (θ-Δθ<θ'<θ+Δθ) or the azimuth angle of the second direction is within a predetermined range from the azimuth angle of the opposite direction of the first direction (-θ-Δθ<θ'<-θ+Δθ) and the elevation angle of the second direction is within a predetermined range from the elevation angle of the first direction (φ-Δφ<φ'<φ+Δφ). If it is judged that the conditions are satisfied (step S204 YES), the process proceeds to step S206. If it is judged that the conditions are not satisfied (step S204 NO), the process proceeds to step S208.

[0062] (Step S206) The masking analysis unit 116 calculates the masking amount M of the band components of the essential object by the band components of the non-essential object in the m-th band using the formula (1). m (dB) is calculated, and then the process proceeds to step S210. (Step S208) The masking analysis unit 116 calculates the masking amount M of the band components of the essential object by the band components of the non-essential object in the m-th band using the formula (2). m (dB) is calculated, and then the process proceeds to step S210. (Step S210) The masking analysis unit 116 calculates the level s of the band component of the essential object in the m-th band. m and the masking amount M m Level difference with m -M m Then, the process of FIG. 3 ends.

[0063] In the above embodiment, the gain for the band components of the audio of the non-essential object depends on the presence or absence of the audio of the essential object and the level of the band components of the essential object in that frequency band. Moreover, the dependency may differ depending on the frequency band. As the gain varies over time, the level of the band components of the non-essential object varies. This level variation may cause discomfort to the listener.

[0064] Therefore, the volume adjustment unit 118 may mitigate the fluctuation of the level determined at each predetermined time interval. The volume adjustment unit 118 may, for example, calculate a moving average value of the level up to the latest section up to that point in time, and output the calculated moving average value to the output processing unit 122. The moving average may be either a simple moving average or an exponential moving average. The volume adjustment unit 118 may continuously change the level from the level in the previous section to the level in the latest section over a predetermined time (for example, a time interval related to the level calculation).

[0065] In the above embodiment, the voicedness determination unit 114 determines whether the voice of the mandatory object is present, the masking amount by the non-mandatory object's voice and the level difference with the mandatory object's voice are calculated in the masking analysis unit 116, and the volume adjustment of the non-mandatory object by the volume adjustment unit 118 are performed for each frequency band. However, this is not limiting. These processes may be performed collectively for all frequency bands. In this case, the band division of the audio signal in the masking analysis unit 116 and the output processing unit 122 and the synthesis process between frequency bands of band components in the output processing unit 122 are omitted.

[0066] In addition, the masking analysis unit 116 determines the masking amount using a function parameter of one of two stages depending on the angle between the first direction and the second direction, but is not limited to this. The masking analysis unit 116 may determine the masking amount using a function parameter of one of three or more stages depending on the angle between the first direction and the second direction, or may determine the masking amount corresponding to the angle between the first direction and the second direction using a predetermined continuous function.

[0067] The parameters according to this embodiment, such as the reference level difference, the parameters for calculating the masking amount (Equations (1) and (2)), the range of directions from the first direction (i.e., the azimuth angle width Δθ, the elevation angle width Δφ), etc., are not limited to the values ​​exemplified above and may be larger or smaller values. The function for calculating the masking amount from the essential object band components and the non-essential object band components is not limited to the functions shown in formulas (1) and (2), and may be any function capable of estimating the masking amount according to the human hearing characteristics. Also, a function capable of estimating the masking amount in a unified manner regardless of the angle between the first direction and the second direction may be used.

[0068] The voice processing device 10 does not necessarily have to include all of the playback unit 130, the display unit 140, and the input unit 150, as long as the control unit 110 can be connected to the playback unit 130, the display unit 140, and the input unit 150 so that various data can be transmitted and received wirelessly or via wires between the control unit 110 and each of the playback unit 130, the display unit 140, and the input unit 150. Some or all of the playback unit 130, the display unit 140, and the input unit 150 may be omitted from the voice processing device 10.

[0069] In the above description, the audio processing device 10 is capable of presenting 6DoF content, and mainly handles listening positions distributed in a three-dimensional space and sound source positions for each object, but this is not limited thereto. The audio processing device 10 may be capable of presenting at least 4DoF content, and handle listening positions distributed in a two-dimensional space and sound source positions for each object. In this case, among the coordinate values ​​in the three-dimensional space, the radial distance and azimuth angle are adopted, and the elevation angle may be ignored.

[0070] As described above, the audio processing device 10 of this embodiment includes an output processing unit 122 that renders audio for each object according to the sound source direction based on the listening position, an analysis unit (e.g., masking analysis unit 116) that calculates the amount of masking of the audio of the second type of object by the audio of the first type of object so that the amount of masking decreases as the angle between a first direction, which is the sound source direction of a first type of object (e.g., a required object) among the objects, and a second direction, which is the sound source direction of a second type of object (e.g., a non-required object) increases, and an adjustment unit (e.g., volume adjustment unit 118) that adjusts the level of the audio of the second type of object so that, when the level difference between the level of the audio of the first type of object and the masking amount is smaller than a predetermined reference level difference, the level difference becomes equal to or greater than a predetermined reference level difference. According to this configuration, when the angle between the first direction and the second direction becomes smaller, the amount of masking by the second type object increases, and the level difference between the level of the sound of the first type object and the amount of masking by the second type object becomes smaller than the reference level difference, the level of the sound of the second type object is lowered so that the level difference becomes equal to or greater than the reference level difference. Therefore, in a situation where the sound of the second type object is reproduced, the sound of the first type object can be clearly heard by the listener.

[0071] The analysis unit may also calculate the amount of masking based on the smaller of a first angle, which is the angle between the first direction and the second direction, and a second angle, which is the angle between the second direction and an inversion direction that is a direction inverted forward and backward from the first direction. According to this configuration, the smaller of the first and second angles is the angle between the left and right directions obtained without distinguishing between the front and rear of the first and second directions, and the masking amount is calculated based on that angle. Therefore, even if one or both of the sound of the first object and the second object are perceived in a direction that is inverted from the target direction, a masking amount according to the angle between the perceived directions can be obtained.

[0072] In addition, when there are multiple second type objects, the adjustment unit may calculate a weighted sum of the individual masking amounts for the audio of each second type object as the masking amount for the entire second type object, using a weighting coefficient that is smaller the larger the angle between the first angle and the second angle, and adjust the audio level for each second type object based on the level difference between the audio level of the first type object and the masking amount. According to this configuration, the masking amount of the entire second type object is calculated so that the contribution of the second type object is smaller as the angle between the second direction and the first direction becomes larger. Since a value closer to the actual measurement value is calculated than when the contributions of the multiple second type objects are uniform, the sound level of the second type object is adjusted based on the calculated value, and the sound of the first type object can be clearly heard without lowering the sound level of the second type object more than necessary.

[0073] In addition, the analysis unit may calculate an amount of masking of the sound of the first type object by the sound of the second type object for each predetermined frequency band, and the adjustment unit may adjust the level of the sound of the second type object based on a level difference between the level of the sound of the first type object and the calculated amount of masking. According to this configuration, the masking amount is calculated for each frequency band, and the level of the sound of the second type object is adjusted based on the level difference between the sound level of the first type object and the calculated masking amount. Therefore, the sound level of the second type object is adjusted taking into consideration the presence or absence of band components of the first type object for each frequency band and the characteristics of masking that depend on the frequency band.

[0074] In addition, all or part of the above-mentioned voice processing device 10, for example, the content acquisition unit 112, the voice presence determination unit 114, the masking analysis unit 116, the volume adjustment unit 118, and the output processing unit 122, may be realized by a computer. The functions may be realized by recording a program for realizing the functions in a computer-readable recording medium, and reading and executing the program recorded in the recording medium into a computer system. In addition, the "computer system" here refers to a computer system built into the voice processing device 10, and includes hardware such as an OS (Operating System) and peripheral devices. In addition, the "computer-readable recording medium" refers to portable media such as a flexible disk, an optical magnetic disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built into a computer system. Furthermore, the "computer-readable recording medium" may also include a device that dynamically holds a program for a short time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a device that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Furthermore, the above program may be for realizing part of the functions described above, and may further be capable of realizing the functions described above in combination with a program already recorded in the computer system.

[0075] In addition, a part or the whole of the voice processing device 10 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the voice processing device 10 may be individually processed, or a part or the whole may be integrated and processed. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0076] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0077] 10: voice processing device, 110: control unit, 112: content acquisition unit, 114: voice presence determination unit, 116: masking analysis unit, 118: volume adjustment unit, 122: output processing unit, 130: playback unit, 140: display unit, 150: input unit

Claims

1. an output processing unit that renders sound for each object according to a sound source direction based on a listening position; an analysis unit that calculates a masking amount of the sound of the first type object by the sound of the second type object so that the masking amount decreases as an angle between a first direction that is a sound source direction of a first type object and a second direction that is a sound source direction of a second type object increases; when a level difference between the audio level of the first object and the masking amount is smaller than a predetermined reference level difference; an adjustment unit that adjusts the level of the audio of the second type object so that the level difference is equal to or greater than a predetermined reference level difference. Audio processing device.

2. The analysis unit includes: a first angle which is an angle between the first direction and the second direction, and a second angle which is an angle between the second direction and an inverted direction which is a direction inverted forward and backward with respect to the first direction, Calculate the masking amount based on the smaller angle The audio processing device according to claim 1 .

3. The adjustment unit is when the number of the second type objects is more than one, a weighted sum of individual masking amounts of the first type object for each of the second type objects is calculated as the masking amount of the entire second type objects by using a weighting coefficient that is smaller as the angle is larger; adjusting the audio level for each of the second objects based on the level difference The audio processing device according to claim 1 .

4. The analysis unit calculates the amount of masking for each predetermined frequency band, The adjustment unit adjusts a level of the sound of the second object based on a level difference between a level of the sound of the first object and the amount of masking for each of the frequency bands. The audio processing device according to claim 1 .

5. To your computer A program for causing the voice processing device according to claim 1 to function.

Images