Audio signal processing device and program thereof

Abstract

To provide an audio signal processing device and a program thereof capable of providing playback sound that reflects the production intent.SOLUTION: An audio signal adjustment unit acquires an audio signal whose level has been adjusted according to the distance from the listening position to the target position for each audio object, and adjusts the level of the audio signal of at least one of the audio objects such that the audio object with a higher priority has a higher priority. This embodiment can also be realized as an audio signal processing device or a program for causing a computer to function as the audio signal processing device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present application relates to an audio signal processing device and a program thereof, for example, a technology for adjusting the playback level according to the priority of each audio object in 6DoF (Degrees of Freedom) content that allows a listener to set the listening position or listening direction of an audio object. [Background technology]

[0002] In recent years, audio technologies that support 6DoF content (see Non-Patent Documents 6 and 7) have been developed by extending object-based audio systems (see Non-Patent Documents 3-5) that can manipulate audio content composed of audio signals with attached audio metadata (see Non-Patent Documents 1 and 2). 6DoF refers to the degree of freedom of an object's movement in six directions in a three-dimensional space. In 6DoF content, the listener sets an arbitrary position and orientation, and the content listened to at the set position and orientation is simulated. It differs from conventional 3D (Dimensional) audio in that the acoustic characteristics of the reproduced sound change depending on the set position or orientation. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] ITU-R BS.2076-1, Audio Definition Model, June 2017 [Non-Patent Document 2] ITU-R BS.2125-0, A serial representation of the Audio Definition Model, January 2019 [Non-Patent Document 3] ISO / IEC 23008-3:2019, Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3 3D audio, 2019 [Non-Patent Document 4] ETSI TS 103 190-2, Digital Audio Compression (AC-4) Standard; Part2: immersive and personalized audio, V1.2.1,2018-02 [Non-Patent Document 5] ATSC Standard: A / 342:2021 Part 3, MPEG-H System, 11 March 2021 [Non-Patent Document 6] MPEG-I Immersive Audio Encoder Input Format, Version 5, April 4, 2023 [Non-Patent Document 7] Report ITU-R BT.2420-5, Collection of usage scenarios of advanced immersive sensory media systems (09 / 2022) Summary of the Invention [Problem to be solved by the invention]

[0004] In general, the goal of audio and sound production is to optimize the auditory sensation at a predefined listening position. In 6DoF content, the listening position is not fixed. This makes it difficult to control the quality of the reproduced sound that is heard. For example, when there is a line that the producer wants the listener to hear as an audio object, if the listener is far away from the speaker, the audio level of the line will decrease. As a result, the listener may miss the line. In other words, because the listener is given freedom, the reproduced sound may deviate from the production intention.

[0005] The embodiments of the present application have been made to solve the above-mentioned problems, and one of the objects of the invention is to provide an audio signal processing device and a program therefor that can provide reproduced sound that reflects the creator's intentions. [Means for solving the problem]

[0006] [1] One aspect of this embodiment is an audio signal processing device that includes an audio signal adjustment unit that acquires an audio signal whose level is adjusted for each audio object according to the distance from the listening position to a target position, and adjusts the level of the audio signal of the audio object so that the level of the audio signal is higher for audio objects with higher priority. According to the configuration of [1], even if the level of the audio signal is adjusted according to the distance from the listening position, the level of the audio signal of the audio object is determined in descending order of priority, so that the higher the priority of the audio object, the easier it is for the listener to perceive it. Therefore, the production intention regarding the audio object that should be preferentially listened to can be reflected in the playback of the playback sound.

[0007] [2] One aspect of this embodiment is the above-mentioned audio signal processing device, wherein the audio signal adjustment unit may determine an adjustment amount for the level of the audio signal of a first object, which is an audio object to be processed, and a second object, which is an adjacent audio object with a lower priority than the first object, so that the level difference between the first object and a second object is equal to or greater than a predetermined level difference. According to the configuration of [2], the level difference between adjacent audio objects with priority is equal to or greater than a predetermined level difference. Since the level difference between audio objects can be easily identified by a listener, the priority between the audio objects can be intuitively recognized.

[0008] [3] One aspect of this embodiment is the above-mentioned audio signal processing device, wherein the audio signal adjustment unit may determine the adjustment amount so that the difference in average loudness levels between the first object and the second object is a multiple of a predetermined level difference of the priority of the first object. According to the configuration of [3], the lower the priority of an audio object, the greater the difference in loudness level between it and an adjacent audio object with a lower priority. Therefore, even if an audio object with a lower priority has a lower perceived volume, the difference in volume between it and other audio objects can be easily identified.

[0009] [4] One aspect of this embodiment is the above-mentioned audio signal processing device, wherein the audio signal adjustment unit may determine the validity of each audio object, exclude invalid audio objects, and re-set the priority of valid audio objects. According to the configuration of [4], the level of the audio signal is set for the audio objects that are active at each point in time. By omitting the level setting for the inactive audio objects, it is possible to avoid unnecessary reduction in the level for the audio objects with lower priority.

[0010] [5] One aspect of this embodiment is the audio signal processing device described above, wherein when the audio signal adjustment unit stops adjusting the level of the audio signal of the second object, the amount of change in the level from the reference level before the adjustment may be eliminated from the adjustment amount over a certain period of time or more. According to the configuration of [5], when the adjustment of the audio signal level is stopped, the level that was lowered by the adjustment is increased over a certain period of time, thereby preventing the listener from feeling uncomfortable due to a sudden increase in level.

[0011] [6] One aspect of the present embodiment may be a program for causing a computer to function as the above-described audio signal processing device. According to the configuration of [6], even if the level of the audio signal is adjusted according to the distance from the listening position, the level of the audio signal is determined in descending order of priority for the audio objects, so that the higher the priority of the audio object is, the easier it is for the listener to perceive it. Therefore, the production intention regarding the audio object that should be preferentially listened to can be reflected in the playback of the playback sound. Effect of the Invention

[0012] According to the above configuration, it is possible to provide a reproduced sound that reflects the creator's intentions. [Brief description of the drawings]

[0013] [Figure 1] 1 is an explanatory diagram for explaining an outline of an audio signal processing device according to an embodiment of the present invention; [Diagram 2] 1 is a schematic block diagram showing an example of the configuration of a voice processing system according to an embodiment of the present invention; [Diagram 3] 2 is a schematic block diagram showing an example of a functional configuration of an audio signal processing unit according to the present embodiment. FIG. [Figure 4] 4 is a schematic block diagram showing an example of a functional configuration of an audio signal adjustment unit according to the present embodiment. FIG. [Diagram 5] 5 is a flowchart illustrating an example of audio signal processing according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (overview) Hereinafter, an embodiment of the present application will be described with reference to the drawings. First, an outline of the present embodiment will be described. Fig. 1 is an explanatory diagram for explaining the outline of an audio signal processing device 10 according to the present embodiment.

[0015] The audio signal processing device 10 according to the present embodiment acquires an audio signal whose acoustic characteristics have been adjusted for each audio object according to the positional relationship with the listening position, and adjusts the level of the audio signal of the audio object so that the higher the priority of the audio object, the higher the level of the audio signal of the audio object. In adjusting the level of the audio signal, the level of the audio signal of the lower object is adjusted so that the level difference between the audio object to be processed (sometimes referred to as a "higher object" in this application) and an adjacent audio object (sometimes referred to as a "lower object" in this application) having a lower priority than the audio object to be processed (sometimes referred to as a "higher object" in this application) is equal to or greater than a predetermined level difference.

[0016] In the example of Fig. 1, audio signals of three audio objects forming audio content are input to the audio signal processing device 10. The audio content may be provided independently or may be provided accompanying other media type content, for example, video content. The audio content may be audio included in any genre of content, such as drama, documentary, variety show, movie, animation, etc.

[0017] The three audio objects include dialogue, sound effects, and background music (BGM), and priorities are set in that order. In this example, dialogue has the highest priority, and background music has the lowest priority. Priorities are set in advance by the producer, and are not changed by the user, the listener. However, the audio signal processing device 10 may exclude from the priority order an audio object that is not provided with an audio signal higher than a predetermined level, and adjust the level of each audio signal based on the priorities determined among the other audio objects. The audio signal processing device 10 determines that an audio object for which no priority order is set has the lowest priority.

[0018] The reproduced sound obtained by the audio signal processing device 10 contains audio signal components of valid audio objects. A valid audio object is an audio object to which an audio signal of a certain level or higher is provided, i.e., an active audio object. An invalid audio object to which an audio signal of a certain level or higher is not provided, i.e., an inactive audio object, may be excluded (skipped) from the priority setting targets. If the timing at which an audio signal is provided differs for each audio object, the effective priority of each audio object for the entire audio content fluctuates over time. As a result, the level of the audio signal of a valid audio object fluctuates.

[0019] In the example in Figure 1, time t 0 From 1 Period until t 4 From 5 Period until t 10 From 11 During these periods, only the audio signal of BGM is provided. During these periods, a predetermined reference level is set for the audio signal of BGM, which has the lowest priority, without level adjustment taking into consideration other audio objects. time t 1 From 2 Period until t 5 From 6 Period until t 7 From 8 During the period up to , the audio signal of the dialogue is provided in addition to the BGM, and the audio signal of the sound effect is not provided. During these periods, the priority of the actual audio objects is dialogue, then BGM. Therefore, the level of the audio signal of the dialogue is higher than the level of the audio signal of the BGM.

[0020] time t 3 From 4 Period until t 9 From 10During the period up to , in addition to the BGM, the audio signal of the sound effect, which has a higher priority, is provided, and the audio signal of the dialogue is not provided. During these periods, the priority of the actual audio objects is sound effects, followed by BGM. Therefore, the level of the audio signal of the sound effect is higher than the level of the audio signal of the BGM. time t 2 From 3 Period until t 6 From 7 Period until t 8 From 9 During the period up to , the audio signals of the dialogue, sound effects, and background music are all provided. During these periods, the priority order of the actual audio objects is dialogue, sound effects, background music, and sound effects. Therefore, the audio level of the dialogue is the highest, and the level of the audio signal for each audio object is lower in the order of dialogue, sound effects, and background music.

[0021] When any valid audio object becomes invalid, the level adjustment is stopped for the lower-level object adjacent to the invalid audio object with a lower priority. In that case, the level of the lower-level object may increase suddenly, which may cause discomfort to the listener. Therefore, the audio signal processing device 10 may cancel the amount of change in the level from the level before the adjustment from the adjustment amount used for the level adjustment over a certain period of time from the point of stopping the level adjustment. For example, at a time t 3 At time t , the dialogue is no longer provided with a certain level of audio signal or higher, and the dialogue is invalidated. At that time, the effective priority of the sound effect changes from second to first, and the audio signal processing device 10 cancels the level adjustment for the sound effect. 3 Then, over a certain period of time, the amount of change in level from the level before adjustment is reduced from the adjustment amount at the time of adjustment to 0. This reduces the increase in sound effect level over time. The audio signal processing device 10 determines the level for BGM, which has a lower priority, by having a predetermined level difference from the level for sound effects. Therefore, the increase in the BGM level is also reduced. This reduces or eliminates the discomfort felt by the listener due to a sudden increase in the level of the audio signal.

[0022] Next, a configuration example of the voice processing system 1 according to this embodiment will be described. FIG. 2 is a schematic block diagram showing a configuration example of the voice processing system 1 according to this embodiment. The voice processing system 1 is capable of adjusting the level of the voice signal for two or more voice objects. In general, the number of voice objects may differ depending on the content, but the following description will mainly focus on the case where the number of voice objects is three. In addition, the functional configuration related to a specific voice object may be distinguished by using a sub-number such as "-1" or "-2". When there is no need to distinguish between these, the sub-number is omitted. Unless otherwise specified, the functional configuration of each voice object is common between voice objects.

[0023] The audio processing system 1 includes an audio signal processing device 10 and a playback device 20. The audio signal processing device 10 makes it possible to set target positions for sound image localization with respect to a listening position for multiple audio objects constituting an audio content. The audio signal processing device 10 adjusts the level according to the target position set for the audio signal acquired for each audio object (rendering). The audio signal processing device 10 adjusts the level of the audio signal based on the individual priority for each audio object. The audio signal processing device 10 generates data including the adjusted level for each audio object, acoustic metadata indicating the target position, and the audio signal as playback data indicating the playback sound, and outputs the data to the playback device 20. The playback device 20 presents the playback sound perceived from different target positions with the level adjusted for each audio object based on the playback data input from the audio signal processing device 10.

[0024] Next, a description will be given of an example of the functional configuration of the audio signal processing device 10. The audio signal processing device 10 includes a position information setting unit 112, a rendering unit 114, and an audio signal processing unit 120. The position information setting unit 112 sets user position information relating to the listening position of the user. When setting the user position information, the position information setting unit 112 may display, for example, a setting screen showing a listening space including the user's head and its surroundings on a display unit (not shown). The position information setting unit 112 may determine, as a listening position, a position corresponding to a position in the listening space designated by an operation signal input from an input unit (not shown), and determine, as a listening direction, a direction of the head designated by the operation signal. The position information setting unit 112 may acquire a position detection signal indicating the detected position from a position sensor (not shown) that detects the position of the user. The position information setting unit 112 may acquire a direction detection signal indicating the detected direction from a direction sensor (not shown) that detects the direction of the user's head. In addition, the position information setting unit 112 may display a sign (for example, an icon, a button, a figure, etc.) representing an audio object in the listening space of the setting screen, and determine a target position corresponding to the position of the sign designated by the operation signal. The position information setting unit 112 outputs target position information indicating the determined target position to the rendering unit 114.

[0025] The rendering unit 114 sets a target position based on the listening position for the audio signal acquired for each audio object forming the audio content, adjusts the volume according to the set target position (rendering), and generates an audio signal for each object. 1 ~o 3 is input to the rendering unit 114. 1 ,o 2 ,o 3 include the audio signals of dialogue, sound effects, and background music, respectively. In this application, the audio object refers not only to the sound source itself that can be operated, but also to a concept including metadata related to the sound source (sometimes referred to as "acoustic metadata" in this application).

[0026] The rendering unit 114 includes a position determination unit 114a and a distance attenuation determination unit 114b. The position determination unit 114a determines a target position of an audio object with the listening position indicated in the user position information input from the position information setting unit 112 as a reference position (for example, the origin). When audio metadata is provided for each audio object and a target position is set in the audio metadata, the position determination unit 114a adopts the target position. The position determination unit 114a may adopt the target position of the audio object indicated in the target position information input from the position information setting unit 112. The target position is expressed by the distance and direction from the listening position. The position determination unit 114a notifies the distance attenuation determination unit 114b of the determined target position.

[0027] The distance attenuation determination unit 114b determines the amount of attenuation for the audio signal based on a predetermined distance attenuation law, based on the distance indicated by the target position notified by the position determination unit 114a. For example, the inverse square law is used as the distance attenuation law. The rendering unit 114 attenuates the amplitude of the audio signal acquired for each audio object according to the attenuation amount determined by the distance attenuation determination unit 114b. The rendering unit 114 includes the target position determined by the position determination unit 114a as audio metadata of the audio object. The rendering unit 114 may calculate an average loudness level for each predetermined period (e.g., several hundred ms to several seconds) as an index of the volume level of the audio signal after attenuation. The rendering unit 114 may include the calculated average loudness level as audio metadata of the audio object.

[0028] The rendering unit 114 associates the audio signal, the level of which has been adjusted for each audio object, with the audio metadata and outputs the result to the audio signal processing unit 120. In the example of FIG. 1 ~o 3 The processed audio objects o 1 '~o 3 ' is output to the audio signal processor 120.

[0029] The audio signal processing unit 120 adjusts the level of the audio signal of an audio object input from the rendering unit 114 so that the higher the priority of the audio object, the higher the level of the audio signal of the audio object. The audio signal processing unit 120 generates data including the acoustic metadata and the audio signal whose level has been adjusted for each audio object as playback data indicating the playback sound. The audio signal processing unit 120 outputs the generated playback data to the playback device 20. An example of the functional configuration of the audio signal processing unit 120 will be described later.

[0030] The playback device 20 plays back the playback sound based on the playback data input from the audio signal processing unit 120. The playback device 20 generally has a plurality of playback channels, and is provided with a playback sound source (for example, headphones or speakers) for each playback channel. The playback device 20 separates audio metadata and audio signals for each audio object from the playback data. The playback device 20 generates an audio signal for each object by adding an acoustic characteristic corresponding to a target direction indicated by the audio metadata to the audio signal whose level has been adjusted for each audio object. The playback device 20 adds (mixes) the generated audio signals for each object between audio objects for each playback channel to generate a playback signal. The playback sound source of the playback device 20 emits a playback sound based on the generated playback signal.

[0031] The process for adding the acoustic characteristics depends on the audio reproduction method of the reproduction device 20. When the playback method is multi-channel audio, playback processing information indicating a reference direction for each playback sound source and a gain for each channel corresponding to a target direction is set in the playback device 20. By referring to the playback processing information, the playback device 20 can identify two or more playback channels that spatially include a target direction for an audio object and identify a gain corresponding to the reference direction for each playback channel. The playback device 20 can generate a playback signal whose amplitude is adjusted by multiplying the acquired audio signal by the gain identified for each playback channel.

[0032] When the playback method is binaural sound, playback processing information indicating a head-related transfer function corresponding to a target direction for each playback channel is set in advance in the playback device 20. By referring to the playback processing information, the playback device 20 can specify a head-related transfer function corresponding to a target direction for an audio object for each playback channel, and convolve the specified head-related transfer function with the acquired audio signal to generate a playback signal.

[0033] Next, an example of the functional configuration of the audio signal processing unit 120 according to the present embodiment will be described. 3 is a schematic block diagram showing an example of the functional configuration of the audio signal processing unit 120 according to this embodiment. The audio signal processing unit 120 includes a priority order determination unit 122, an audio signal adjustment unit 124, an audio multiplexing unit 126, and an audio metadata multiplexing unit 128.

[0034] The priority determination unit 122 refers to the priority metadata for each audio object constituting the audio content and specifies the priority of the audio object. The priority metadata may be provided to the priority determination unit 122 as part of the audio metadata in association with the audio signal for each audio object, or the audio object may be provided separately. The priority metadata may be provided, for example, in association with a video object or an avatar constituting a series of content including the audio object. The priority determination unit 122 may determine the priority of an audio object for which priority metadata has not been acquired as the lowest, and generate priority metadata indicating the lowest priority. The priority determination unit 122 outputs the audio metadata including the priority metadata acquired for each audio object to the audio signal adjustment unit 124 together with the audio signal. A voice signal and a sound object for each voice object are input to the voice signal adjustment unit 124. The voice signal adjustment unit 124 adjusts the level of the voice signal so that the higher the priority of the voice object, the higher the level of the voice signal.

[0035] The audio signal adjustment unit 124 processes audio objects with higher priority, and selects the top object o to be processed. i' (i is an integer between 1 and the number of voice objects, indicating the priority) and the adjacent lower object o with a lower priority than the upper object i+1 ' so that the level difference between the lower object o and the i+1 The audio signal adjustment unit 124 adjusts the level of the audio signal of the higher-level object o′, for example, as shown in equation (1). i 's average loudness level L i and the subordinate object o i+1 's average loudness level L i+1 The difference between the two is a predetermined level difference of 10 l (l is a positive real number greater than 0) i ' to make the average loudness level L i+1 Adjustment value α for i+1 [dB]. The audio signal adjustment unit 124 calculates the adjustment value α i+1 Subtract o to get the lower object i+1 Audio signal for 's i+1 Adjusts the average loudness level of the

[0036]

number

[0037] The audio signal adjustment unit 124 may determine the validity of each audio object. For example, the audio signal adjustment unit 124 determines an audio object whose average loudness level is equal to or higher than a predetermined lower limit as valid, and determines an audio object whose average loudness level is less than the predetermined lower limit as invalid. When the acoustic metadata or a series of contents includes a validity flag indicating whether the audio object is valid or not (ON / OFF), the audio signal adjustment unit 124 may determine whether the audio object is valid or not according to the validity flag. The audio signal adjustment unit 124 excludes (skips) audio objects determined to be invalid, and adjusts the priority levels identified for audio objects determined to be valid so that the priority levels differ by 1. The audio signal adjustment unit 124 may include priority level metadata indicating the adjusted priority levels for each audio object determined to be valid in the audio metadata.

[0038] When k-1 (k is an integer greater than 1) invalid voice objects (i.e., skipped voice objects) are included among the lower objects due to the priority adjustment, the relationship shown in formula (1) is expressed by formula (2). In formula (2), j is the number of voice objects L i j is an integer equal to or greater than 0 that indicates the number of audio objects whose levels have been adjusted before the upper object o is processed. i The actual priority of the average loudness level L i+k According to equation (2), the adjustment value α i+1 Instead of, the adjustment value β i+k is calculated. The adjustment value β i+k According to the subordinate object o i+k Audio signal for 's i+k The average loudness level of the

[0039]

number

[0040] The audio signal adjustment unit 124 adjusts the average loudness level difference shown in equation (1), that is, 20log 10 An upper limit (for example, 15 to 25 dB) may be set in advance for i+l. When the difference in the average loudness level exceeds the upper limit, the audio signal adjustment unit 124 uses the upper limit instead of the average loudness level to adjust the adjustment value α i+1 Similarly, the audio signal adjustment unit 124 may determine the difference in average loudness level shown in formula (2), that is, 20log 10If the difference in the average loudness levels exceeds the upper limit, the audio signal adjustment unit 124 uses the upper limit instead of the average loudness levels to adjust the adjustment value β for the level of the audio signal of the adjacent audio object. i+k may be determined.

[0041] In the example of FIG. 3, the audio signal adjustment unit 124-1 adjusts the audio object o 1 ' as the upper object, and the voice object o 2 ' is the lower object, and the average loudness level of the upper object is L 1 and the average loudness level of the lower objects L 2 Based on the adjustment value α for the subordinate object 2 The audio signal adjustment unit 124-1 calculates the adjustment value α 2 The audio signal s of the lower object is subtracted from 2 's average loudness level and the resulting audio signal s 2 " to the audio multiplexing unit 126-1. The audio signal adjustment unit 124-1 adjusts the subordinate object o 2 's acoustic metadata m 2 In the upper object o, the metadata indicating the original average loudness level is replaced with metadata indicating the adjusted average loudness level. 1 's acoustic metadata m 1 ' and the acoustic metadata m obtained by replacing 2 " to the audio metadata multiplexing unit 128-1.

[0042] The audio multiplexing unit 126-1 multiplexes the audio signal s of the superior object input from the audio signal adjustment unit 124-1. 1 ' and the audio signal of the lower object s 2 " and the multiplexed audio signal s obtained by the integration 1 '+s 2 " to the audio metadata multiplexing unit 128-1. The audio metadata multiplexing unit 128-1 multiplexes the upper object o input from the audio signal adjustment unit 124-1.1 's acoustic metadata m 1 to the subordinate object o 2 Acoustic metadata m relating to ' 2 " and multiplexed audio metadata m 2 " is generated. The audio metadata multiplexing unit 128-1 multiplexes the lower object o 2 The multiplexed audio signal s generated as 1 '+s 2 " and the multiplexed audio metadata m 2 " and outputs the result to the audio signal adjustment unit 124-2.

[0043] The audio signal adjustment unit 124-2 adjusts the audio object o 2 ' to the upper object, the voice object o 3 ' is the lower object, and the average loudness level L of the upper object is 2 and the average loudness level of the lower objects L 3 Based on the adjustment value α for the subordinate object 3 The audio signal adjustment unit 124-2 calculates the adjustment value α 3 The audio signal s of the lower-level object input from the priority order determination unit 122 is obtained by subtracting 3 The audio signal adjustment unit 124-2 adjusts the average loudness level of the audio signal s' of the lower-level object obtained by adjusting the average loudness level. 3 " to the audio multiplexing unit 126-2. The audio signal adjustment unit 124-2 adjusts the subordinate object o 3 's acoustic metadata m 3 In the upper object o, the metadata indicating the original average loudness level is replaced with metadata indicating the adjusted average loudness level. 2 's acoustic metadata m 2 " and the subordinate object o obtained by replacement 3 's acoustic metadata m 3 " to the audio metadata multiplexing unit 128-2.

[0044] The audio multiplexer 126-2 multiplexes the multiplexed audio signal s input from the audio signal adjuster 124-2. 1 '+s 2 " and the subordinate object o 3 Audio signal for 's 3 " and the multiplexed audio signal s obtained by the integration 1 '+s 2 "+s 3 " to the audio metadata multiplexing unit 128-2. The audio metadata multiplexing unit 128-2 receives the higher-level object o input from the audio signal adjustment unit 124-2. 2 's acoustic metadata m 2 " to the lower object o 3 Acoustic metadata m relating to ' 3 " and multiplexed audio metadata m 3 " is generated. The audio metadata multiplexing unit 128-1 multiplexes the lower object o 3 The multiplexed audio signal s generated as 1 '+s 2 "+s 3 " and the multiplexed audio metadata m 3 " is associated with the playback device 20.

[0045] In the example of FIG. 1, the number of valid voice objects is three. More generally, the number of final voice objects to be played back is three. n The reproduced data includes multiplexed audio signals s 1 '+Σ i=2 n s i " where n indicates the number of valid audio objects. The playback data includes acoustic metadata for each of the n audio objects. The acoustic metadata conveys the average loudness level of each of the audio objects with priority 2 or lower.

[0046] Next, an example of the functional configuration of the audio signal adjustment unit 124-i according to this embodiment will be described. FIG. 4 is a schematic block diagram showing an example of the functional configuration of the audio signal adjustment unit 124-i according to the present embodiment. The audio signal adjustment unit 124-i adjusts the audio object o i ' is a speech object that is at the level and subordinate to i+1 ' so that the level difference between the audio objects o and o is equal to or greater than a specified level difference. i+1 Audio signal for 's i+1 Adjust the level of '. The audio signal adjustment unit 124-i includes audio metadata separation units 124a-i and 124a-(i+1), an adjustment level determination unit 124b, an adjustment timing setting unit 124c, a level adjustment unit 124d, and an audio metadata resetting unit 124e.

[0047] The audio metadata separation unit 124a-i receives a higher-level object from the audio metadata multiplexing unit 128-i (FIG. 3). However, when i=1, the higher-level object is received from the priority order determination unit 122. The audio metadata separation unit 124a-i extracts an audio signal s from the higher-level object. i ' and the acoustic metadata m i ' and the separated audio signal s i The audio metadata separation unit 124a-i outputs the separated audio metadata m i ' to the adjustment level determination unit 124b.

[0048] The audio metadata separator 124a-(i+1) receives the lower level objects from the priority order determination unit 122 (FIG. 3). The audio metadata separator 124a-(i+1) extracts the audio signal s from the lower level objects. i+1 ' and the acoustic metadata m i+1 ' and the separated audio signal s i+1 The audio metadata separation unit 124a-(i+1) outputs the separated audio metadata m i+1 ' to the adjustment level determination unit 124b and the acoustic metadata resetting unit 124e.

[0049] The adjustment level determination unit 124b receives the higher-level object o from the audio metadata separation unit 124a-i. i 's acoustic metadata m i ' is input, and the audio metadata separator 124a-(i+1) outputs the lower object o i+1 's acoustic metadata m i+1 The adjustment level determination unit 124b receives the acoustic metadata m i The average loudness level L i and the acoustic metadata of the lower object m i+1 Average loudness level L indicated by ' i+1 Based on this, for example, the average loudness level L is calculated using equation (1). i+1 Adjustment amount a for i+1 The adjustment level determination unit 124b calculates the calculated adjustment amount a i+1 The level adjustment unit 124d and the audio metadata resetting unit 124e output adjustment level metadata indicating the above.

[0050] The adjustment timing setting unit 124c is configured to set the upper object o input from the audio metadata separation unit 124a-i. i Audio signal for 's i 'Based on the subordinate object o i+1 Audio signal for 's i+1 Adjustment timing metadata that indicates when to adjust the level of i+1 Generate.

[0051] For example, when no level adjustment is performed, the adjustment timing setting unit 124c adjusts the audio signal s every frame of a predetermined length (for example, every tens to hundreds of ms). i The adjustment timing setting unit 124c measures the level of the audio signal, and judges whether the frame is a voice frame or a silent frame depending on whether the measured level is equal to or higher than a predetermined lower limit. The adjustment timing setting unit 124c judges the start of level adjustment when the number of voice frames continues for a predetermined number of consecutive frames or more. The adjustment timing setting unit 124c receives adjustment timing metadata t i+1 to the level adjustment unit 124d. The adjustment timing setting unit 124c determines the end of the level adjustment when a silent frame continues for a predetermined number of consecutive frames or more during the level adjustment. i+1 to the level adjustment unit 124d. i+1 can also be regarded as switching information indicating whether or not a level adjustment is required.

[0052] The level adjustment unit 124d adjusts the adjustment timing metadata t i+1 Based on the adjustment level metadata input from the adjustment level determination unit 124b, the lower-level object o input from the audio metadata separation unit 124a-(i+1) is i+1 Audio signal for 's i+1 The level adjustment unit 124d adjusts the level of the adjustment timing metadata t i+1 When the average loudness level L indicates the need for level adjustment 1+1 Adjustment amount α indicated in adjustment level metadata i+1 The audio signal s is subtracted i+1 The level adjustment unit 124d starts adjusting the level of the adjustment timing metadata t i+1 When the level is not adjusted, the average loudness level L 1+1 Adjustment amount α for i+1 A speech signal based on s i+1 The level adjustment of the audio signal s' is completed. i+1 ' to the audio multiplexing unit 126-i.

[0053] The acoustic metadata resetting unit 124e resets the subordinate object o input from the acoustic metadata separation unit 124a-(i+1). i+1 's acoustic metadata m i+1 Average loudness level L indicated by ' i+1 From the adjustment level metadata α input from the adjustment level determination unit 124b i+1 By subtracting the average loudness level L i+1The acoustic metadata resetting unit 124e adjusts the acoustic metadata m i+1 Average loudness level before adjustment, indicated by 'L i+1 Instead, the adjusted average loudness level L i+1 The acoustic metadata resetting unit 124e resets the acoustic metadata m i+1 ' to the audio metadata multiplexing unit 128-1 (FIG. 3).

[0054] In addition, when the content to be provided includes a validity flag for each audio object, the adjustment timing setting unit 124c may determine the start or end of level adjustment based on the validity flag instead of or together with the audio signal. For example, the adjustment timing setting unit 124c determines the start of level adjustment when a validity flag indicating validity is acquired for an audio object designated as invalid. The adjustment timing setting unit 124c determines the end of level adjustment when a validity flag indicating invalidity is acquired for an audio object designated as valid. When a validity flag indicating a start time at which an audio object designated as invalid is acquired, the adjustment timing setting unit 124c may determine the start of level adjustment at the start time. When a validity flag indicating an end time at which an audio object designated as valid is acquired, the adjustment timing setting unit 124c may determine the end of level adjustment at the end time.

[0055] The level adjustment unit 124d uses the adjustment timing metadata t i+1 The amount of change from the audio signal level (reference level) at the start of level adjustment over a predetermined time (attack time) from the start of level adjustment varies from 0 to an adjustment amount α i+1 The level adjustment unit 124d may change the adjustment timing metadata t i+1 The level difference between the reference level and the audio signal level at the end of the level adjustment is adjusted by an amount of adjustment α over a predetermined time (release time) or more after the end of the level adjustment.i+1 The attack time may be changed continuously from 0 to 0. Therefore, the increase in the level of the audio signal is gradual. The attack time may be smaller than the release time. The attack time may be 0 ms. The release time is adjusted by the adjustment amount α i+1 The release time may be longer as the value of i is larger. The release time may be, for example, an integer multiple of i or j in formulas (1) and (2). An upper limit (for example, 1 second) may be set for the release time, limiting longer release times.

[0056] Next, an example of audio signal processing according to this embodiment will be described. Fig. 5 is a flowchart showing an example of audio signal processing according to this embodiment. The processing in Fig. 5 is repeatedly executed at regular intervals (time windows, for example, several hundred ms to several seconds). (Step S102) The audio signal processing unit 120 acquires an audio signal and acoustic metadata for each audio object from the rendering unit 114. (Step S104) The priority order determining unit 122 obtains priority order metadata for each audio object. (Step S106) The priority order determination unit 122 determines the priority order for each audio object based on the acquired priority order metadata. The audio signal adjustment unit 124 sets the audio object with the highest priority order as the initial value of the higher-order object to be processed.

[0057] (Step S108) The audio signal adjustment unit 124 judges whether or not the superior object is valid based on the audio signal. If it is judged to be valid (step S108 YES), the process proceeds to step S112. If it is judged to be invalid (step S108 NO), the process proceeds to step S110. (Step S110) The audio signal adjustment unit 124 changes the higher-level object to an adjacent audio object with a lower priority, and then proceeds to the process of step S108.

[0058] (Step S112) The audio signal adjustment unit 124 judges whether or not a lower-level object adjacent to the higher-level object with a lower priority is valid based on the audio signal. If it is judged to be valid (step S112 YES), the process proceeds to step S118. If it is judged to be invalid (step S112 NO), the process proceeds to step S114. (Step S114) The audio signal adjustment unit 124 judges whether or not an audio object with a lower priority exists. If it is judged that an audio object with a lower priority exists (step S114 YES), the process proceeds to step S116. If it is judged that an audio object with a lower priority does not exist (step S114 NO), the process proceeds to step S132. (Step S116) The audio signal adjustment unit 124 changes the lower object to an adjacent audio object with a lower priority (skip), and then proceeds to the process of step S112.

[0059] (Step S118) The audio signal adjustment unit 124 determines the adjustment amount of the audio signal level of the lower object so that the difference between the level of the audio signal of the higher object and the level of the audio signal of the lower object becomes a predetermined level difference. (Step S120) The audio signal adjustment unit 124 determines the timing for adjusting the level of the audio signal of the lower object based on the change in the level of the audio signal of the upper object. (Step S122) When it is determined that the level adjustment is necessary, the audio signal adjustment unit 124 adjusts the level of the audio signal of the lower object based on the determined adjustment amount.

[0060] (Step S124) The audio signal adjustment unit 124 adjusts the level indicated by the acoustic metadata of the lower object based on the calculated adjustment amount. The acoustic metadata resetting unit 124e resets the level after adjustment to replace the level before adjustment indicated by the acoustic metadata of the lower object.

[0061] (Step S126) The audio multiplexing unit 126 updates the multiplexed audio signal by integrating and multiplexing the level-adjusted audio signal into a multiplexed audio signal including audio signals for each audio object lower than the priority of the higher-level object. (Step S128) The audio metadata multiplexing unit 128 multiplexes the adjusted audio metadata of the lower object into the playback data including the multiplexed audio signal and audio metadata for each audio object lower than the priority of the higher-level object.

[0062] (Step S130) The audio signal adjustment unit 124 judges whether or not there is an audio object with a lower priority than the lower object. If it is judged that there is an audio object with a lower priority than the lower object (step S130 YES), the audio signal adjustment unit 124 updates the lower object at that time to a higher object and returns to the processing of step S112. If it is judged that there is no audio object with a lower priority than the lower object (step S130 NO), the processing proceeds to step S132. (Step S132) The audio metadata multiplexing unit 128 outputs the playback data to the playback device 20, and reproduces the playback sound. After that, the process in FIG. 5 ends.

[0063] 5. In this case, no level adjustment is performed, and playback data is obtained including the audio signal for each audio object input from the rendering unit 114 and the audio object, and is output to the playback device 20. The audio signal processing device 10 may further include a playback device 20 and may be configured as a single audio signal processing device 10. In the audio signal processing device 10, the position information setting unit 112 and the rendering unit 114 may be omitted, and an audio signal and an audio object for each rendered audio object may be provided.

[0064] Note that N (N is an integer of 3 or more) audio objects included in the content provided to the audio signal processing device 10 may be classified into K (K is an integer of 2 or more and smaller than N) groups. Each group may be distinguished according to the type of audio object, and a priority order may be set. The groups may be called classes or layers. The types of audio objects include, for example, dialogue (spoken voice such as lines), sound effects, BGM (Background Music), and the like. Therefore, the audio signal adjustment unit 124 may adjust the level for each group so that the level difference between groups with adjacent priorities is equal to or greater than a predetermined level difference. The audio signal adjustment unit 124 may adjust the level of the audio signal of each audio object for each group so that a predetermined level is obtained.

[0065] In the above description, the audio processing system 1 is mainly assumed to be capable of reproducing 6DoF audio content and to handle listening positions distributed in a three-dimensional space and target positions for each audio object, but this is not limited thereto. The audio processing system 1 may be capable of reproducing at least 4DoF audio content and to handle listening positions distributed in a two-dimensional space and target positions for each audio object.

[0066] As described above, the audio signal processing device 10 according to the present embodiment acquires an audio signal whose level has been adjusted according to the distance from the listening position to the target position for each audio object, The audio signal adjustment unit 124 adjusts the level of the audio signal of an audio object so that the higher the priority of the audio object, the higher the priority of the audio object. According to this configuration, even if the level of the audio signal is adjusted according to the distance from the listening position, the level of the audio signal of the audio object is determined in descending order of priority, so that the higher the priority of the audio object, the easier it is for the listener to perceive it. Therefore, the production intention regarding the audio object that should be preferentially listened to can be reflected in the reproduction of the reproduced sound.

[0067] In addition, the audio signal adjustment unit 124 may determine the amount of adjustment for the level of the audio signal of a first object (e.g., a higher-level object), which is an audio object to be processed, so that the level difference between the first object (e.g., a higher-level object), which is an adjacent audio object with a lower priority than the first object, is greater than or equal to a predetermined level difference. According to this configuration, the level difference between audio objects having adjacent priorities is equal to or greater than a predetermined level difference. Since the level difference between audio objects can be easily identified by a listener, the priorities between the audio objects can be intuitively recognized.

[0068] Furthermore, the audio signal adjustment unit 124 may determine the adjustment amount so that the difference in average loudness level between the first object and the second object is a multiple of the priority level of the first object of a predetermined level difference. According to this configuration, the lower the priority of an audio object, the greater the difference in loudness level between the audio object and an adjacent audio object with a lower priority. Therefore, even if an audio object with a lower priority has a smaller perceived volume, the difference in volume between the audio object and other audio objects can be easily identified.

[0069] Also, the audio signal adjuster 124 may determine the validity of each audio object, exclude invalid audio objects, and reset the priority order of valid audio objects. According to this configuration, the level of the audio signal is set for each audio object that is valid at each point in time. By omitting level setting for invalid audio objects, it is possible to avoid unnecessary reduction in the level for audio objects with lower priorities.

[0070] In addition, when the audio signal adjustment unit stops adjusting the level of the audio signal of the second object, the audio signal adjustment unit may cancel the amount of change in the level from the level before the adjustment from the set adjustment amount over a certain period of time (e.g., a release time) or more. According to this configuration, when the adjustment of the audio signal level is stopped, the level that was lowered by the adjustment is increased over a certain period of time, thereby making it possible to avoid an unpleasant feeling felt by the listener due to a sudden increase in level.

[0071] All or part of the above-mentioned audio signal processing device 10, for example, the position information setting unit 112, the rendering unit 114, and the audio signal processing unit 120, may be realized by a computer. The functions may be realized by recording a program for realizing the functions in a computer-readable recording medium, and reading and executing the program recorded in the recording medium into a computer system. The term "computer system" as used herein refers to a computer system built into the audio signal processing device 10, and includes hardware such as an OS (Operating System) and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above-mentioned program may be a program for realizing a part of the above-mentioned functions, or may be a program that can be realized by combining the above-mentioned functions with a program already recorded in the computer system.

[0072] Also, a part or all of the audio signal processing device 10 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the audio signal processing device 10 may be individually processed, or a part or all of the blocks may be integrated and processed. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, when an integrated circuit technology that can replace LSI appears due to the advancement of semiconductor technology, an integrated circuit based on that technology may be used.

[0073] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0074] 1...audio processing system, 10...audio signal processing device, 20...playback device, 112...position information setting unit, 114...rendering unit, 120...audio signal processing unit, 122...priority determination unit, 124...audio signal adjustment unit, 124a-i, 124a-(i+1)...acoustic metadata separation unit, 124b...adjustment level determination unit, 124c...adjustment timing setting unit, 124d...level adjustment unit, 124e...acoustic metadata resetting unit

Claims

1. Acquire an audio signal whose level is adjusted according to the distance from the listening position to the target position for each audio object; and an audio signal adjusting unit that adjusts the level of an audio signal of the audio object so that the audio object has a higher priority. Audio signal processing device.

2. The audio signal adjustment unit is An adjustment amount of the level of the audio signal of a first object, which is an audio object to be processed, and a second object, which is an adjacent audio object having a lower priority than the first object, is determined so that the level difference between the first object and the second object is equal to or greater than a predetermined level difference. The audio signal processing device according to claim 1 .

3. The audio signal adjustment unit is The adjustment amount is determined so that a difference between average loudness levels of the first object and the second object is a multiple of a predetermined level difference and a priority level of the first object. The audio signal processing device according to claim 2 .

4. The audio signal adjustment unit is Determine the validity of each audio object; Filter out invalid audio objects and reprioritize valid ones The audio signal processing device according to claim 1 .

5. The audio signal adjustment unit is When the adjustment of the level of the audio signal of the second object is stopped, The amount of change in the level from the level before the adjustment is eliminated from the amount of adjustment over a certain period of time or more. The audio signal processing device according to claim 2 .

6. A program for causing a computer to function as the audio signal processing device according to claim 1.

Images